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PREFACE 
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also  provided  valuable  advice  for  clarifying  arguments  and  strengthening  the  document.  And 
Barb  Law  and  Teresa  Heller  provided  useful  assistance  in  making  necessary  arrangements  and  in 
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SUMMARY 


This  report  explores  the  implications  of  the  Naturalistic  Decision  Making  framework  for 
the  domain  of  Information  Dominance,  which  has  been  defined  as  an  operational  advantage 
obtained  through  superior  effectiveness  of  informational  activity.  Naturalistic  Decision  Making 
(NDM)  is  the  study  of  how  people  use  their  experience  to  make  decisions  in  field  settings.  The 
expertise  Klein  Associates  considered  was  at  both  the  individual  and  the  team  level  of  decision 
making.  We  defined  what  expertise  consists  of  and  we  identified  some  important  barriers  that 
might  be  particularly  troublesome  in  an  era  of  Information  Dominance.  These  barriers  include 
the  following: 

excessive  data 
pre-processed  data 
excessive  procedures 
performing  formal  analyses 

passive  data  handling,  limited  ability  for  information  seeking 
interfaces  that  obscure  the  big  picture 

The  danger  exists  that  these  barriers  will  severely  limit  the  use  of  expertise  at  the 
individual  and  the  team  level,  and  that  information  technology  will  result  in  reduced  rather  than 
increased  performance. 

The  challenge  is  to  understand  how  information  technologies  can  interfere  with  expertise, 
and  to  develop  procedures  for  ensuring  that  these  technologies  support  proficiency  in  carrying 
out  Information  Dominance  functions. 

The  NDM  perspective  can  offer  some  ideas  about  using  information  technology  to 
support  skilled  decision  making.  We  have  identified  several  possible  directions  for  future 
research  into  learning  more  about  the  barriers  themselves:  predicting  the  extent  to  which  new 
designs  will  increase  or  decrease  performance  levels,  developing  new  system  designs  that  will 
support  the  key  requirements  of  decision  makers,  and  defining  the  affordances  of  Information 
Dominance  for  expanding  situation  awareness  and  for  decreasing  the  adversary’s  situation 
awareness. 
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I.  INTRODUCTION 


The  rapid  developments  of  information  technology  are  generating  a  wide  range  of 
implications  for  the  way  military  leaders  conceptualize  future  warfare.  One  of  the  emerging 
themes  in  future  operations  is  Information  Dominance.  Information  Dominance  covers  a  number 
of  different  fields,  including  hardware  and  software  engineering,  communications  technology, 
sensor  technology,  artificial  intelligence,  encryption  techniques,  C4I,  and  has  been  defined  in 
many  different  ways.  Whitaker  and  Kuperman  (1996)  have  recently  examined  many  of  the 
cognitive  engineering  and  human  factors  aspects  of  Information  Dominance  and  for  the  purpose 
of  this  report  we  will  use  their  definition  of  Information  Dominance  which  is  as  follows: 

Information  Dominance  is  an  operational  advantage  obtained  through 
superior  effectiveness  of  informational  activity  (acquisition  and  processing  of 
data,  information,  and/or  knowledge),  to  the  extent  that  this  advantage  is 
demonstrable  through  superior  effectiveness  of  instrumental  activity. 

The  purpose  of  this  report  is  to  explore  the  implications  of  the  Naturalistic  Decision 
Making  framework  for  the  domain  of  Information  Dominance.  Inasmuch  as  Information 
Dominance  can  be  conceptualized  as  a  way  of  making  better  and  faster  decisions,  while 
degrading  and  slowing  down  the  decision  cycle  of  an  adversary,  new  insights  can  be  gained  by 
viewing  Information  Dominance  from  a  decision  making  point  of  view. 

The  organization  of  the  report  is  as  follows.  Section  II  describes  the  Naturalistic 
Decision  Making  framework,  and  Section  III  presents  a  recognitional  model  of  Naturalistic 
Decision  Making.  Section  IV  discusses  the  aspects  of  expertise  found  in  decision  making. 

Section  V  is  a  critical  discussion  of  the  barriers  that  information  technology  poses  to  proficient 
decision  making,  and  a  set  of  hypotheses  about  the  ways  that  Information  Dominance  programs 
can  make  these  barriers  more  severe,  resulting  in  worse  performance  rather  than  increased 
performance.  Then  in  Section  VI  we  shift  to  an  examination  of  team  decision  making, 
particularly  the  concept  of  the  “team  mind.”  Section  VII  discusses  the  nature  of  expertise  in  team 
decision  making.  Section  VIII  examines  the  barriers  to  proficient  team  decision  making.  Section 
IX  offers  recommendations  for  shaping  information  technologies  to  support  proficient  individual 
and  team  decision  making  and  an  agenda  for  future  research.  Appendix  A  presents  a  review  of 
the  literature  on  the  cognitive  aspects  of  team  decision  making. 


II.  WHAT  IS  NATURALISTIC  DECISION  MAKING? 

Naturalistic  Decision  Making  (NDM)  can  be  defined  as  the  study  of  how  people  use  their 
experience  to  make  decisions  in  field  settings.  The  NDM  framework  (Flin,  1996;  Flin,  Martin, 
Strub,  &  Salas,  in  press;  Klein,  Orasanu,  Calderwood,  and  Zsambok,  1993;  Zsambok  and  Klein, 
1997)  is  designed  to  investigate  the  strategies  people  use  in  performing  complex,  ill-structured. 


1 


high-stakes  tasks,  under  time  pressure  and  uncertainty,  with  changing  conditions,  and  in  the 
context  of  team  and  organizational  constraints. 

The  NDM  framework  arose  independently  of  the  normative  decision  research  tradition. 
The  normative  research  tradition  assumes  the  decision  maker  is  a  rational/economic  person 
(Simon,  1956)  or  vigilant  decision  maker  (Janis  &  Mann,  1977)  who  systematically  searches  for 
relevant  information  in  an  unbiased  manner  and  then  carefully  weighs  the  utility  of  each 
alternative  before  making  a  choice.  This  tradition  is  also  referred  to  as  the  Rational  Choice 
Model  of  decision  making.  This  approach  evaluates  the  quality  of  decision-making  behavior  by 
comparing  the  decision-maker’s  performance  to  a  normative  statistical  model,  such  as  Bayes 
Theorem,  to  determine  whether  the  optimal  choice  was  made. 

In  the  period  1985-1989,  a  number  of  researchers  studied  the  way  people  made  judgments 
and  decisions  in  realistic  settings  such  as  urban  fire  suppression,  identification  of  hostile  units  in 
military  engagements,  and  nuclear  power  plant  operation.  Many  of  these  efforts  had  been  funded 
by  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences,  in  an  attempt  to 
design  training  and  decision  support  systems  that  could  be  put  into  use.  And  most  of  the 
researchers  were  grounded  in  other  domains  (e.g.,  engineering,  training,  human  factors)  with 
little  or  no  background  in  the  normative  decision  research  tradition.  The  research  methods  were 
primarily  descriptive,  aimed  at  gaining  an  initial  understanding  of  the  way  that  experienced 
personnel  handled  the  difficulties  of  time  pressure,  uncertainty,  and  the  other  factors  listed  above. 
The  Army  Research  Institute  sponsored  a  conference  on  NDM  in  1989,  and  this  may  be 
considered  the  beginning  of  the  NDM  movement.  At  this  conference  (Klein,  Orasanu, 
Calderwood,  &  Zsambok  1993),  the  investigators  learned  that  they  were  observing  the  same 
processes  even  though  they  were  studying  different  domains. 

What  are  these  common  processes?  Researchers  working  in  the  NDM  framework  have 
observed  that  for  most  high-stakes  tasks,  people  with  experience  are  making  the  decisions,  rather 
than  novices.  Furthermore,  experience  enables  decision  makers  to  identify  reasonable  courses  of 
action  as  the  first  ones  considered  (Klein,  Wolf,  Militello,  &  Zsambok,  1995),  so  that  the  burden 
of  difficulty  is  on  assessing  the  nature  of  the  situation  rather  than  on  comparing  alternative 
courses  of  action.  In  addition,  in  most  settings  it  is  less  important  to  arrive  at  the  optimal  choice 
than  it  is  to  arrive  at  a  workable  choice  in  a  short  period  of  time  —  a  tactic  Simon  (1956)  has 
called  “satisficing.”  Uncertainty  is  a  common  difficulty  in  naturalistic  settings.  However,  people 
do  not  appear  to  try  to  estimate  the  level  of  uncertainty  for  different  outcomes,  or  to  quantify  the 
uncertainty  for  different  hypotheses.  Instead,  experienced  decision  makers  have  developed 
strategies  for  managing  uncertainty  (Schmitt  &  Klein,  1996),  which  can  involve  knowing  when 
to  accept  it,  when  to  try  to  seek  more  information,  and  how  to  structure  a  situation  in  order  to 
reduce  the  uncertainty.  Cohen,  Freeman,  and  Wolf  (1996)  have  discussed  the  importance  of 
metacognitive  strategies  in  managing  uncertainty. 

Lipshitz  (1993)  reviewed  nine  different  NDM  models,  and  identified  a  set  of  common 
themes.  These  included  the  importance  of  situation  awareness,  the  use  of  mental  imagery  in 
many  of  the  models,  the  importance  of  context  in  trying  to  understand  real-world  decisions,  the 
shifts  between  intuitive  and  analytical  strategies,  and  the  emphasis  on  helping  people  by 
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supporting  the  decision  strategies  they  use  rather  than  by  trying  to  have  them  adopt  different 
decision  strategies. 

Zsambok  (1997)  has  identified  several  themes  that  characterize  NDM  research.  First,  in 
the  period  from  1989  to  1994  she  found  activity  in  developing  and  testing  models.  Second,  the 
focus  of  NDM  is  expanding  to  include  a  greater  interest  in  related  lines  of  work,  such  as  studies 
of  expertise,  problem  solving,  situation  awareness,  and  process  control.  Third,  is  the  growing 
emphasis  on  situation  awareness  as  a  key  aspect  of  decision  making.  Fourth,  is  the  effort  that 
goes  into  improving  the  methodologies  for  carrying  out  field  research.  Fifth,  is  the  interest  in 
finding  applications  for  NDM  findings. 

Klein  (1997a)  has  discussed  some  trends  in  the  applications  growing  out  of  the  NDM 
framework.  On  the  surface,  it  would  appear  that  NDM  research  should  have  less  application  than 
normative  decision  research,  because  NDM  seeks  to  describe  what  people  do,  whereas  normative 
studies  often  try  to  discover  deviations  from  optimal  strategies  in  order  to  prescribe  better 
strategies.  However,  as  was  discussed  above,  the  normative  strategies  cannot  be  used  in  many 
field  settings  because  the  boundary  conditions  in  terms  of  data  quality  and  time  available  to 
perform  the  analyses  are  not  met. 

The  NDM  framework  cannot  rely  on  normative  standards  for  evaluating  performance,  but 
it  can  use  the  strategies  of  skilled  decision  makers  to  serve  as  criteria  for  evaluating  novices. 
Furthermore,  a  naturalistic  approach  will  attempt  to  build  on  the  strategies  people  already  use, 
rather  than  seeking  to  replace  these  strategies.  As  a  result,  the  likelihood  of  acceptance  is  greater, 
because  resistance  is  reduced. 

One  final  advantage  of  the  NDM  framework  is  that  it  tends  to  ground  the  applications 
within  a  context.  Whereas  the  normative,  analytical  approach  had  the  strength  of  being  generic, 
it  had  the  weakness  of  not  being  grounded  within  the  context  of  a  specific  domain.  Thus,  a 
strategy  for  comparing  the  strengths  and  weaknesses  of  options  can  be  used  in  selecting 
automobiles,  buying  homes,  choosing  a  school  for  a  child,  but  it  does  not  have  the  depth  of  a 
strategy  used  by  baggage  screeners  in  airports  to  identify  threatening  items  to  inspect  more 
closely,  or  a  strategy  used  by  air  traffic  controllers  to  spot  early  warning  signs  that  airplanes  may 
soon  be  violating  separation  criteria.  These  latter  strategies  are  domain  specific,  and  context- 
restricted.  In  supporting  a  new  baggage  screener  or  a  new  air  traffic  controller,  it  may  be  more 
useful  to  consider  the  context-specific  strategies  used  by  the  experienced  decision  makers. 
Teaching  multi-attribute  utility  analysis  methods  or  Bayesian  statistics  to  air  traffic  controllers 
does  not  appear  to  have  great  value. 


III.  THE  RECOGNITION-PRIMED  DECISION  MODEL 

The  Recognition-Primed  Decision  (RPD)  model  is  one  of  the  examples  of  how  NDM 
research  can  bring  a  new  perspective  to  understanding  how  people  use  their  experience  to  make 
decisions  in  field  settings.  The  RPD  model  was  originally  developed  to  explain  how  fireground 
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commanders  were  able  to  use  their  experience  to  select  a  course  of  action  without  having  to 
compare  different  options.  Fireground  commanders  are  the  leaders  of  teams  that  are  called  out 
whenever  there  is  a  fire  or  related  emergency,  primarily  in  urban  settings.  The  model  was  based 
on  interviews  and  observations  of  fireground  commanders  working  with  difficult  and  challenging 
incidents.  Klein,  Calderwood,  and  Clinton-Cirocco  (1986)  examined  more  than  30  incidents. 

The  commanders  were  working  against  severe  time  pressure,  since  more  than  80%  of  the 
decisions  were  made  in  less  than  a  minute.  The  stakes  were  high;  poor  decisions  could  result  in 
loss  of  lives  and  property.  Information  quality  was  uneven,  goals  shifted,  and  conditions 
changed.  The  situational  dynamics  changed  an  average  of  five  times  in  each  incident  studied. 
Klein  et  al.  expected  that  the  commanders  would  have  to  resort  to  a  limited  comparison  between 
options,  and  were  surprised  to  discover  that  the  commanders  reported  they  were  not  making  any 
comparisons  at  all. 

These  findings  raised  two  key  questions:  First,  how  could  the  commanders  be  sure  of 
carrying  out  effective  courses  of  action  without  generating  a  set  of  options  from  which  to 
choose?  Second,  how  could  the  commanders  evaluate  a  course  of  action  without  comparing  it  to 
others?  Klein  et  al.  (1986)  carefully  examined  the  interview  data  and  the  156  decision  points 
probed,  and  developed  the  RPD  model  based  on  the  fireground  commanders’  own  accounts. 

The  answer  to  the  first  question  (how  the  commanders  did  not  have  to  generate  a  set  of 
options)  was  that  the  commanders  could  use  their  experience  to  size  up  a  situation  and  thereby 
recognize  the  typical  action  to  take.  They  could  generate  a  reasonable  option  as  the  first  one 
considered.  They  were  not  trying  to  find  the  optimal  solution,  but  rather  to  quickly  arrive  at  a 
workable  solution  that  could  be  enacted  in  time  to  arrest  the  spread  of  a  fire  that  might  be 
growing  exponentially. 

The  answer  to  the  second  question  (how  the  commanders  could  evaluate  an  option 
without  comparing  it  to  others)  was  that  once  the  commanders  identified  a  typical  course  of 
action,  they  would  evaluate  it  by  imagining  it,  mentally  simulating  it  to  see  if  it  would  work  in 
the  context  of  the  situation  they  were  facing.  If  the  course  of  action  was  found  satisfactory,  it 
would  be  initiated  without  any  further  delay.  If  they  found  any  flaws,  they  would  switch  to  a 
problem-solving  mode  to  repair  the  flaws.  If  they  could  not  repair  the  flaws,  they  would  reject 
the  course  of  action  and  consider  the  next  most  typical  reaction,  repeating  the  process  until  they 
found  a  workable  option. 

In  short,  the  firefighters  were  able  to  use  their  experience  to  identify  a  workable  course  of 
action  as  the  first  one  they  considered.  If  they  needed  to  evaluate  a  course  of  action,  they 
conducted  a  mental  simulation  to  see  if  it  would  work. 
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Figure  1:  Recognition-Primed  Decision  model. 

The  RPD  model  is  shown  in  Figure  1 .  The  simplest  case  is  where  a  decision  maker  sizes 
up  a  situation,  forms  expectancies  about  what  is  going  to  happen  next,  determines  the  cues  that 
are  most  relevant,  recognizes  the  reasonable  goals  to  pursue  in  the  situation,  recognizes  a  typical 
reaction,  and  carries  it  out.  This  is  probably  also  the  most  common  case.  We  consider  this  a 
decision  because  reasonable  alternative  courses  of  action  could  have  been  taken.  Other  decision 
makers,  perhaps  with  less  experience,  might  have  selected  these  alternatives.  Therefore,  a 
decision  point  hypothetically  existed  even  though  the  decision  maker  did  not  treat  it  as  such. 

The  second  panel  of  Figure  1  shows  a  more  difficult  case,  where  the  decision  maker  is 
not  certain  about  the  nature  of  the  situation.  Perhaps  some  anomaly  arises  that  violates 
expectancies  and  forces  the  decision  maker  to  question  whether  the  situation  is  perhaps  different 
than  it  seems.  Another  possibility  is  that  the  uncertainty  might  be  present  from  the  beginning. 
Here,  decision  makers  must  deliberate  about  what  is  happening.  Studying  the  Commanding 
Officers  and  Tactical  Action  Officers  of  AEGIS  cruisers,  Kaempf,  Wolf,  Thordsen,  and  Klein 
(1996)  found  that  one  strategy  they  use  is  to  build  a  story  that  explains  the  various  pieces  of 
information.  If  there  are  competing  interpretations  of  the  situation,  the  decision  maker  may  try  to 
build  a  story  for  each,  and  appraise  which  story  is  the  most  consistent  and  plausible.  Diagnosis  is 
the  attempt  to  link  the  observed  events  to  causal  factors;  by  establishing  such  a  linkage  the 
decision  maker  would  obtain  an  explanation  for  the  events.  Diagnosis  is  important  for  the  RPD 
model  because  the  nature  of  the  situation  can  largely  determine  the  course  of  action  adopted. 
Often,  decision  makers  will  spend  more  time  and  energy  trying  to  determine  what  is  happening, 
and  distinguishing  between  different  explanations,  than  comparing  different  courses  of  actions. 
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Diagnostic  activity  is  initiated  in  response  to  uncertainty  about  the  nature  of  the  situation. 
The  purpose  of  the  diagnosis  is  either  to  evaluate  an  uncertain  assessment  of  the  situation  or  to 
compare  alternative  explanations  of  events.  Two  common  diagnostic  strategies  are  feature 
matching  and  story  building.  In  their  study  of  Navy  anti-air  warfare  incidents,  Kaempf  et  al. 
(1996)  examined  103  instances  of  diagnosis.  Of  these,  the  most  common  strategy  was  to  use 
feature  matching  to  assign  a  diagnosis.  Feature  matching  consists  of  identifying  the  relevant 
features  of  a  situation  in  order  to  categorize  it.  This  occurred  in  87%  of  the  cases.  In  12%  of  the 
cases  the  decision  maker  engaged  in  story  building  to  accomplish  the  diagnosis  (e.g.,  showing 
that  the  erratic  course  of  an  unidentified  aircraft  could  be  explained  as  a  lost  helicopter  trying  to 
locate  the  carrier  from  which  it  was  launched).  Despite  this  small  proportion,  the  episodes  of 
story  building  were,  in  several  incidents,  the  key  part  of  the  decision-making  activity. 

Story  building  often  involves  a  type  of  mental  simulation  in  which  a  person  attempts  to 
synthesize  the  features  of  a  situation  into  a  causal  explanation  which  can  be  evaluated  and  used 
in  a  number  of  ways.  Pennington  and  Hastie  (1993)  have  also  examined  the  various  functions  of 
stories,  one  of  which  is  to  build  causal  explanations.  Mental  simulation  can  be  used  to  project  a 
course  of  action  forward  in  time  (V ariation  3  in  Figure  1),  and  it  also  can  be  used  to  look 
backwards  in  time  as  a  way  of  making  sense  of  events  and  observations.  Here,  the  decision 
maker  is  trying  to  find  the  most  plausible  story,  or  sequence  of  events  in  order  to  understand 
what  is  going  on,  a  process  of  diagnosis  that  is  intended  to  result  in  situation  awareness. 

The  third  panel  of  Figure  1  shows  that  once  decision  makers  arrive  at  an  understanding  of 
a  situation,  they  will  recognize  a  typical  course  of  action  and  then  evaluate  it  by  mentally 
simulating  what  will  happen  when  they  carry  out  the  action.  In  this  way,  if  they  spot  weaknesses 
in  their  plan,  they  can  repair  the  weaknesses  and  improve  the  plan.  This  is  a  better  strategy  than 
generating  a  large  set  of  options  and  comparing  these  to  find  the  best  one.  The  evaluation  that 
uses  mental  simulation  can  produce  a  better  course  of  action  instead  of  settling  for  picking  one 
from  a  set. 

The  three  panels  are  presented  for  purposes  of  explanation.  The  model  can  be 
synthesized  into  one  diagram,  shown  in  Figure  2. 

The  RPD  model  claims  that  with  experienced  decision  makers: 

•  The  first  option  they  consider  is  usually  workable  so  they  do  not  have  to  generate  a 
large  set  of  courses  of  action  to  make  sure  of  getting  a  good  one. 

•  Comparing  options  is  not  a  goal.  They  generate  and  evaluate  options  one  at  a  time 
instead  of  comparing  their  advantages  and  disadvantages. 

•  Finding  a  workable  course  of  action  is  a  goal.  They  are  trying  to  find  the  first  option 
that  works,  not  the  best  one. 

•  Evaluating  an  option  occurs  by  imagining  how  it  will  be  carried  out,  not  through 
formal  analyses  and  comparisons. 
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•  Options  can  be  strengthened  by  imagining  the  option  being  carried  out,  spotting 
weaknesses,  and  finding  ways  to  avoid  them. 

•  The  focus  is  on  the  way  they  assess  the  situation  and  judge  it  as  familiar,  not  on 
choosing  between  options. 

•  The  emphasis  is  on  being  poised  to  act  quickly,  rather  than  being  paralyzed  until  all 
the  evaluations  have  been  completed. 


Experience  the  Situation  in  a  Changing  Context 


yes,  but 


Implement 
Course  of  Action 


Figure  2:  Synthesized  version  of  the  RPD  model. 

The  RPD  model  asserts  that  people  can  use  experience  to  generate  a  reasonable  course  of 
action  as  the  first  one  considered.  Is  this  a  valid  claim?  Klein,  Wolf,  Militello,  and  Zsambok 
(1995)  tested  this  hypothesis  in  a  study  that  used  16  skilled  chess  players.  With  the  assistance  of 
a  chess  master,  we  selected  four  reasonably  challenging  chess  positions  and  presented  each  of 
these  to  players  whose  official  ratings  were  at  a  “C”  level  (mediocre)  or  an  “A”  level  (very 
strong).  Each  player  was  tested  individually,  and  we  asked  each  one  to  think  aloud  while  trying 
to  generate  a  move  to  play.  The  data  consisted  of  the  first  option  articulated  by  the  player. 

When  the  player  had  repeated  this  process  for  all  four  boards,  we  asked  them  to  rate  the  quality 
of  all  the  legal  moves  on  each  of  the  boards. 

The  most  critical  assertion  of  the  RPD  model  is  that  people  can  use  experience  to 
generate  a  plausible  option  as  the  first  one  they  consider.  If  this  assertion  is  invalid,  the  rationale 
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for  the  RPD  model  disappears. 

Figure  3  shows  the  percentage  of  first  moves  and  all  possible  moves  receiving  each  of 
five  move  quality  ratings.  Figure  3  shows  that  across  the  full  set  of  legal  moves  most  were  rated 
as  poor  ones.  However,  the  very  first  move  that  the  chess  players  considered  was  rated  very 
high,  using  the  players’  own  assessments.  This  means  that,  according  to  their  own  standards,  the 
first  moves  were  good  ones. 


Blunder  Poor  Average  Good  Best 
Move  Quality 


Figure  3:  Quality  ratings  of  first  moves  and  all  legal  moves. 

Still,  we  might  wonder  how  good  these  first  moves  were,  by  objective  standards.  To 
investigate  this  issue,  we  had  used  board  positions  that  were  taken  from  games  analyzed  by  chess 
grandmasters.  The  grandmasters  awarded  points  to  those  options  they  deemed  playable.  Figure 
4  shows  that  the  grandmaster  awarded  points  to  only  20  out  of  124  legal  moves.  If  the 
participants  in  the  study  had  been  selecting  randomly  from  legal  moves,  they  would  have  shown 
the  same  pattern.  However,  they  showed  the  opposite  pattern.  Of  the  64  first  moves  (16 
participants  and  four  board  positions  each),  41  were  moves  that  had  received  grandmaster  points. 
Therefore,  using  objective  criteria,  we  find  that  skilled  decision  makers  can  generate  good 
options  as  the  first  ones  they  consider.  These  findings  confirm  the  claim  made  by  the  RPD 
model  that  people  can  use  experience  to  recognize  typical  actions  that  are  usually  satisfactory. 
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A  second  assertion  of  the  RPD  model  is  that  time  pressure  need  not  cripple  the 
performance  of  decision  makers  who  have  considerable  expertise,  since  they  use  feature 
matching.  Traditional  Rational  Choice  models  would  predict  that  time  pressure  would  be  likely 
to  interfere  with  the  analytical  processes  needed,  and  could  result  in  degraded  performance  at  all 
levels.  An  earlier  study  of  chess  playing  provides  support  for  this  prediction  from  the  model. 
Calderwood,  Klein,  and  Crandall  (1988)  studied  the  quality  of  chess  moves  generated  under 
tournament  conditions,  using  either  regulation  time  (approximately  2.6  minutes  per  move)  or 
blitz  conditions  (approximately  6  seconds  per  move).  We  found  that  even  under  the  extreme 
time  pressure  of  blitz  chess,  move  quality  remained  at  a  very  high  level.  The  rate  of  blunders 
shown  by  class  “B”  players  did  increase,  from  1 1%  to  25%  under  time  pressure,  but  the  rate  of 
blunders  remained  unchanged  for  chess  experts,  8%  vs.  7%. 


Probability  of  selecting  a 
good  move  by  chance 


[ratio  =  1/6] 


Quality  of  the  first  move 
considered 


[ratio  =  4/6] 


Note:  A  move  was  rated  acceptable  if  it  received  points  from  a 
panel  of  grand  masters  and  unacceptable  if  it  did  not.  The 
sixteen  subjects  each  used  four  boards,  for  a  total  of  64  first 
moves.  There  were  approximately  30  legal  first  moves  in  each 
of  the  four  board  positions,  for  a  total  of  124  legal  moves. 


Figure  4:  Objective  evaluation  of  the  first  chess  move  generated. 

A  third  assertion  of  the  model  is  that  experienced  decision  makers  can  adopt  a  course  of 
action  without  comparing  and  contrasting  possible  courses  of  action.  Kaempf  et  al.  (1996) 
probed  78  instances  of  decision  making  in  operational  Navy  anti-air  warfare  incidents  involving 
AEGIS  cruisers.  These  were,  for  the  most  part,  actual  encounters  with  potentially  hostile  forces 
in  which  several  courses  of  action  theoretically  existed.  The  78  instances  were  probed 
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retrospectively,  during  interviews,  to  determine  the  rationale  for  the  decision.  Kaempf  et  al. 
estimated  that  in  78%  of  the  cases  the  decision  maker  adopted  the  course  of  action  without  any 
deliberate  evaluation,  and  in  18%  of  the  cases  the  evaluation  was  accomplished  using  mental 
simulation.  In  only  4%  of  the  cases  was  there  any  introspective  evidence  for  comparisons  of  the 
strengths  and  weaknesses  of  different  options. 

Since  it  was  first  proposed  in  1985,  the  RPD  model  has  received  a  great  deal  of  support 
from  researchers.  Klein  (1989)  has  summarized  the  data  from  studies  with  tank  platoon  leaders, 
design  engineers,  urban  and  wildland  firefighters,  and  brigade  level  military  planners,  showing 
that  the  RPD  model  accounts  for  most  of  the  decision  points  (between  50%  and  95%  of  the 
decisions  made  by  experienced  personnel),  whereas  Rational  Choice  (i.e.,  comparisons  between 
options)  rarely  occurs.  In  a  variety  of  domains,  involving  high  time  pressure  (urban  firefighters) 
and  low  time  pressure  (design  engineers),  individuals  (design  engineers)  and  teams  (wildland 
firefighters,  commercial  aviation  crews),  military  (Army,  Navy)  and  paramilitary  (firefighters), 
and  nonmilitary  (commercial  pilots,  design  engineers),  decision  makers  rarely  use  Rational 
Choice  methods.  Similarly,  Randel,  Pugh,  Reed,  Schuler,  and  Wyman  (1994)  probed  electronic 
warfare  technicians  while  they  were  performing  a  simulated  task,  and  found  that  93%  of  the 
decisions  involved  serial  (i.e.,  non-comparative)  deliberations,  in  accord  with  the  RPD  model. 
Only  two  of  the  38  decisions  studied  were  classified  as  showing  comparisons  between  options. 
Mosier  (1991)  has  obtained  similar  findings  in  a  study  of  pilots  who  were  videotaped  during  a 
simulated  mission.  Pascual  and  Henderson  (1997)  have  also  obtained  data  supporting  the 
prevalence  of  recognitional  decision  making,  in  a  study  of  Army  command-and-control 
personnel.  Driskell,  Salas,  and  Hall  (1994)  found  that  training  experienced  Navy  officers  to 
follow  vigilant  procedures  (e.g.,  systematically  scanning  all  relevant  items  of  evidence  and 
reviewing  the  information  prior  to  making  a  decision)  resulted  in  worse  performance  than  if  the 
Navy  officers  were  allowed  to  use  “hyper- vigilant”  procedures  that  were  compatible  with 
recognitional  decision  making  (e.g.,  scanning  only  the  information  items  needed  to  make  a 
decision,  in  any  sequence,  and  only  reviewing  items  if  necessary). 

Taken  together,  these  studies  show  a  growing  body  of  empirical  support  for  the 
Recognition-Primed  Decision  model  as  a  descriptive  account  of  the  way  experienced  people 
make  decisions. 


Boundary  Conditions 

Klein  (1989)  has  also  speculated  about  the  boundary  conditions  for  using  recognitional 
decision  making  and  those  for  using  Rational  Choice.  These  are  shown  in  Figure  5.  When  time 
is  short,  the  decision  makers  are  experienced,  the  conditions  keep  changing,  and  the  goals  are  ill- 
defined,  a  recognitional  strategy  is  appropriate.  On  the  other  hand,  if  the  decision  makers  are 
novices,  have  ample  time,  have  to  justify  their  choice  to  others,  need  to  resolve  conflicts  among 
team  members  with  different  priorities,  have  to  find  the  best  option,  and/or  are  working  with  a 
task  requiring  a  great  deal  of  computational  complexity,  then  a  Rational  Choice  strategy  makes 
sense.  A  Rational  Choice  strategy  would  be  useful  in  many  operational  settings,  such  as 
selecting  a  weapons  system  during  a  competitive  procurement,  and  prioritizing  research  areas  in 
order  to  allocate  funds. 
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Figure  5:  Boundary  conditions  for  different  decision  strategies. 

It  is  clear  the  NDM  movement  is  not  in  conflict  with  traditional  decision  research. 

Rather,  the  two  approaches  are  addressing  different  questions.  The  traditional  research  is 
studying  how  analytical  methods  can  be  used  to  make  optimal  choices,  whereas  NDM  is  focused 
on  how  people  use  experiential  strategies  to  make  reasonable  choices.  There  will  obviously  be 
times  in  operational  settings  when  people  should  rely  on  analytical  procedures.  There  are  times 
when  people  are  faced  with  the  choice  between  options,  and  attempt  to  select  the  best  one.  NDM 
researchers  do  not  dispute  these  cases.  Instead,  the  NDM  research  shows  that  these  cases  are 
relatively  infrequent,  and  not  necessarily  as  critical  as  decisions  about  the  nature  of  the  situation. 
The  field  of  NDM  needs  to  understand  how  people  use  experiential  strategies  as  well  as 
analytical  strategies,  and  how  they  blend  the  two. 

There  are  other  models  of  naturalistic  decision  making,  such  as  Image  Theory  (Beach, 
1990),  Story  Building  (Pennington  &  Hastie,  1993),  Search  for  Dominance  Structure 
(Montgomery,  1993),  and  the  decision  ladder  (Rasmussen,  1993).  These  are  entirely  consistent 
with  the  RPD  model;  the  differences  pertain  to  emphases.  Image  Theory  is  concerned  with 
explaining  how  decision  makers  incorporate  individual  values  and  expectancies  into  the 
decision-making  process.  Story  building  is  essentially  a  form  of  mental  simulation;  Pennington 
and  Hastie  have  studied  story-building  strategies  in  the  context  of  juror  decision  making,  which 
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is  directed  at  explaining  previous  events  rather  than  projecting  forward  to  anticipate  new  events. 
Search  for  Dominance  Structure  is  an  attempt  to  build  on  observations  by  Soelberg  (1967)  that 
decision  makers  do  not  rationally  contrast  all  options.  Soelberg’ s  research  was  also  a  key 
impetus  for  the  RPD  model,  and  so  both  models  explain  the  same  thing:  the  tendency  of  a 
decision  maker  to  select  an  option  without  comparing  it  to  others.  Montgomery’s  approach  is  to 
envision  a  comparison  between  a  favorite  option  and  a  comparison  option,  aimed  at 
demonstrating  that  the  favorite  option  is  better.  The  RPD  model  goes  further,  and  asserts  that  in 
most  cases  people  do  not  bother  even  going  through  the  motions  of  a  comparison;  they  identify  a 
favorite  and  use  mental  simulation  to  satisfy  themselves  that  this  favorite  will  work. 

Rasmussen’s  decision  ladder  showing  the  linkage  between  skill-based,  rule-based,  and 
knowledge-based  behavior  is  undoubtedly  the  most  well-known  of  NDM  models.  Rasmussen 
describes  a  level  of  skill-based  behavior  that  has  no  analogue  in  the  RPD  model.  Rasmussen’s 
level  of  rule-based  behavior  does  overlap  with  the  RPD  model,  particularly  in  its  simplest  form. 
And  Rasmussen’s  level  of  knowledge-based  behavior  also  overlaps  with  the  RPD  model  in  terms 
of  the  use  of  mental  simulation  to  explain  and  diagnose  events,  and  to  evaluate  courses  of  action. 
The  two  approaches  clearly  are  compatible,  and  are  both  attempts  to  explain  how  people  using 
their  experience  actually  make  decisions. 

We  should  also  identify  further  some  of  the  limitations  of  the  RPD  model.  The  RPD 
model  does  not  address  all  the  concerns  of  naturalistic  decision  making  (e.g.,  the  influence  of 
team  and  organizational  constraints).  Additionally,  the  RPD  model  does  not  reflect  memory  or 
attentional  processes.  It  is  important  to  understand  the  boundary  conditions  of  a  model.  Cohen, 
Freeman,  and  Wolf  (1996)  have  expanded  the  RPD  model  in  order  to  account  for  metacognition 
(the  management  of  thinking)  along  with  decision  making. 


IV.  THE  ROLE  OF  EXPERTISE  IN  NATURALISTIC  DECISION 

MAKING 

One  way  to  understand  the  role  of  expertise  in  NDM  is  to  consider  a  set  of  core  beliefs 
about  expertise  and  decision  making  (Ball  &  Jones,  1996;  Blasiol,  1996;  Jones,  1996;  Baron  & 
Brown,  1991;  Russo  &  Shoemaker,  1989;  Wickelgren,  1938).  These  are  listed  in  Table  1. 

Table  1.  Core  Beliefs  About  Expertise  and  Decision  Making 


•  Intuition  is  something  you’re  bom  with.  Either  you  have  it  or  you  don’t. 

•  The  way  to  make  decisions  is  by  rationally  comparing  options. 

•  The  way  to  solve  problems  is  to  first  define  the  goal,  then  find  ways  to  reduce  the 
difference  between  your  current  and  your  desired  state. 

•  Information  technology  will  eventually  clear  up  the  fog  of  war,  largely  eliminating 
uncertainty. 
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•  Planning  should  be  done  by  figuring  out  how  to  allocate  resources  to  objectives,  then 
building  a  synchronization  matrix. 

•  Situation  awareness  is  achieved  by  starting  with  data,  converting  them  into 
_ information,  then  into  knowledge,  and  converting  knowledge  into  understanding. 

From  the  perspective  of  NDM,  one  of  the  striking  features  of  these  beliefs  is  that  all  of 
them  are  mistaken.  Some  are  only  partially  erroneous,  whereas  others  are  completely  wrong.  By 
examining  these  beliefs,  we  can  gain  a  better  appreciation  of  expertise. 

The  first  belief  asserts  that  intuition  is  something  you  are  bom  with.  A  surprisingly  high 
proportion  of  people  hold  this  belief.  When  an  account  such  as  the  RPD  model  is  described,  one 
common  reaction  is  that  it  might  work  for  the  10%  of  the  decision  makers  who  have  intuition, 
but  the  rest  of  the  population  better  not  depend  on  it.  However,  there  are  no  data  suggesting 
individual  differences  in  intuition.  There  may  be  some  personality  tests  that  distinguish  intuitive 
from  analytical  styles,  but  this  merely  refers  to  the  preferences  in  handling  evidence,  not  to  the 
presence  or  absence  of  intuition  itself.  Within  the  framework  of  NDM,  intuition  is  seen  as  a 
function  of  experience.  It  is  linked  to  perceptual  skills,  which  are  often  difficult  to  articulate. 
Therefore,  decision  makers  may  have  the  experience  of  “intuiting”  a  reaction  because  they 
cannot  explain  the  basis  for  that  reaction.  The  difficulty  is  one  of  articulation  regarding  tacit 
knowledge. 

Bechara,  Damasio,  Tranel,  and  Damasio  (1997)  has  shown  a  physiological  basis  for 
intuition  —  the  physiological  reactions  that  precede  conscious  awareness  of  correct  strategies. 
The  European  tradition  (e.g.,  deGroot,  1986)  has  had  little  difficulty  with  the  concept  of 
intuition,  whereas  in  the  United  States,  we  find  a  much  greater  level  of  suspicion.  If  we  recast 
the  term  “intuition”  in  terms  of  expertise,  the  problems  and  concerns  should  diminish.  The 
pattern  matching  and  perceptual  and  recognitional  skills  are  what  alert  us  to  the  existence  of  a 
problem,  or  an  opportunity. 

Klein  and  Hoffman  (1993)  have  provided  a  perceptual  account  of  expertise  that  helps 
illustrate  the  links  to  intuition.  According  to  this  account,  experienced  decision  makers  see  the 
world  differently  than  novices.  They  can  make  fine  discriminations  that  are  invisible  to  novices. 
They  can  use  mental  simulation  to  recognize  the  precursors  to  a  situation,  and  can  anticipate  the 
next  developments,  whereas  novices  just  see  what  is  in  front  of  them.  Experts  can  recognize 
typicality,  which  is  only  possible  after  a  sufficient  number  of  experiences  to  allow  a  decision 
maker  to  learn  the  characteristics  and  variability  of  situations.  And  by  recognizing  typicality, 
experts  can  quickly  detect  anomalies.  Because  they  are  seeing  a  different  world,  experts  may  not 
realize  that  others  cannot  make  out  what  is  obvious  to  them.  And  because  of  the  perceptual  basis 
of  their  ability,  experts  cannot  describe  it  to  others. 

The  second  belief  asserts  that  the  way  to  make  decisions  is  by  rationally  comparing 
options.  We  have  already  discussed  the  difficulties  with  this  belief  in  the  previous  sections. 
Sometimes  it  makes  sense  to  contrast  options.  But  in  many  operational  settings,  it  is  impractical 
to  follow  this  strategy. 
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The  third  belief  asserts  that  the  way  to  solve  problems  is  to  follow  the  standard  stage 
model  of  defining  the  goal,  then  generating  courses  of  action  for  achieving  the  goal.  The  flaw 
here  is  that  in  most  cases,  we  are  dealing  with  ill-defined  goals.  Therefore,  if  we  must  first  wait 
to  define  the  goal,  we  can  never  begin.  Klein  and  Weitzenfeld  (1978)  have  argued  that  with  ill- 
defined  goals,  problem  solvers  must  simultaneously  be  seeking  solutions  and  redefining  the  goal. 
As  the  solution  attempts  fail,  the  evaluations  will  explain  the  reason  for  the  failure,  and  usually 
this  will  help  clarify  what  goal  properties  are  necessary.  Klein  (in  press)  has  argued  that 
Artificial  Intelligence  (AI)  approaches  to  problem  solving  are  misleading  if  we  take  them  as 
models  of  human  cognitive  processes.  AI  systems  typically  proceed  by  defining  a  large  problem 
space,  composed  of  all  the  pathways  between  the  current  and  desired  state  (thereby  assuming 
well-defined  goals).  The  system  then  conducts  a  search  through  this  problem  space,  either  using 
an  algorithm  or  a  heuristic.  This  account  reduces  problem  solving  to  searching  through  a 
problem  space.  However,  for  most  problems  people  do  not  establish  a  problem  space.  They  are 
redefining  goals,  recognizing  likely  solution  strategies  based  on  the  way  they  represent  the 
problem. 

Klein  and  Wolf  (in  press)  have  argued  that  problem  solving  is  a  constructive  activity, 
rather  than  a  search  through  a  problem  space.  People  appear  to  be  able  to  use  expertise  to 
recognize  leverage  points,  which  are  potential  building  blocks  for  assembling  a  solution.  The  job 
of  the  problem  solver  is  to  pursue  these  leverage  points  to  try  to  construct  a  solution  strategy. 

Another  limitation  of  the  classical  account  of  problem  solving  is  that  it  does  not  provide 
sufficient  emphasis  on  the  process  of  problem  detection.  Klein  and  Crandall  (1997)  have 
analyzed  more  than  50  cases  of  problem  detection,  and  found  that  a  considerable  amount  of 
expertise  is  necessary  to  detect  and  interpret  the  initial  subtle  cues  that  are  the  early  warning 
signs  that  a  problem  is  developing. 

The  fourth  belief  asserts  that  information  technology  will  clear  up  the  fog  of  war.  Even  if 
this  was  true,  it  would  likely  be  irrelevant  because  the  pressure  to  expedite  decision  cycles  would 
merely  result  in  a  situation  where  commanders  were  making  faster  decisions,  at  the  same  level  of 
uncertainty  as  they  were  used  to.  With  the  advent  of  radars  on  ships,  many  expected  that  safety 
would  increase.  In  fact,  ship  captains  had  reached  a  risk  homeostasis,  and  used  the  new 
technology  to  increase  their  speed,  keeping  accident  levels  constant.  So,  too,  with  information 
technology.  If  a  decision  was  formerly  made  in  an  hour,  at  a  level  of  70%  confidence,  and  the 
technology  provided  the  70%  confidence  level  in  30  minutes,  we  suspect  that  the  decision  would 
be  made  at  that  point,  rather  than  waiting  the  full  hour  to  achieve  80%  confidence,  or  waiting 
four  hours  to  achieve  100%  confidence. 

However,  information  technology  is  not  likely  to  clear  up  the  fog  of  war.  Schmitt  and 
Klein  (1996)  have  distinguished  different  sources  of  uncertainty  (missing  information,  unreliable 
information,  ambiguous/conflicting  information,  and  highly  complex  information)  and  different 
levels  of  uncertainty  (data,  inferences  about  the  data,  and  projections  into  the  future).  Thus,  one 
type  of  uncertainty  in  war  is  about  the  location  of  friendly  troops  (missing  data).  Global 
Positioning  System  can  be  helpful  here.  But  another  type  of  uncertainty  is  about  the  intent  of  the 
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enemy  commander  (ambiguous  projection  into  the  future).  Global  Positioning  System  will  not 
help  with  that.  An  historical  mapping  of  enemy  units  might  offer  some  clues.  Another  type  of 
uncertainty  is  the  enemy’s  capability  to  mass  forces  on  a  given  target,  within  a  given  timeframe 
(complex  inference).  And  another  type  of  uncertainty  has  to  do  with  inconsistencies  in  messages 
(unreliable  data).  No  one  technology  will  address  all  of  these  problems.  And  the  increased  pace 
of  communications  will  increase  some  types  of  uncertainties  even  as  it  diminishes  others. 

Skilled  commanders  have  been  able  to  press  on  regardless  of  uncertainty  (e.g.,  Eisenhower 
selecting  a  date  to  cross  the  English  channel  for  D-Day,  despite  uncertain  weather  forecasts),  and 
to  structure  the  battlefield  in  order  to  reduce  uncertainty.  Skilled  commanders  also  know  when 
to  seek  more  information,  whereas  less-talented  commanders  seem  to  be  using  data  collection  as 
a  tactic  to  avoid  making  hard  choices,  looking  for  data  that  are  not  particularly  diagnostic  and 
take  too  long  to  collect.  Expertise  is  therefore  in  the  form  of  when  to  seek  more  information, 
which  information  to  seek,  and  how  to  make  better  use  of  the  available  information. 

The  fifth  belief  is  that  we  should  try  to  achieve  systematic  planning.  Neal  Schmitt  (1997) 
has  described  the  folly  of  aiming  for  complex  synchronization  schedules  in  an  environment  as 
chaotic  as  combat.  Mintzberg  (1994)  has  presented  a  severe  critique  of  the  entire  concept  of 
strategic  planning.  The  arguments  are  essentially  that  the  rigor  is  not  a  substitute  for 
imagination.  Expertise  allows  decision  makers  to  size  up  situations  accurately  and  to  recognize 
leverage  points  as  a  basis  for  constructing  effective  strategies.  Planning  can  be  done  once  the 
strategies  have  been  outlined,  but  strategic  planning  is  not  an  effective  means  of  formulating 
strategy. 

The  sixth  belief  is  that  situation  awareness  is  built  up  from  the  level  of  data,  to  the  level 
of  information,  to  the  level  of  knowledge,  and  finally,  to  the  level  of  understanding.  This  is  an 
extremely  common  belief  (e.g.,  Endsley  &  Robertson,  1996).  It  is  wrong  because  it  ignores  the 
way  experienced  decision  makers  work.  They  do  not  want  to  rely  on  the  analyses  of  less  skilled 
subordinates.  They  do  not  want  to  be  at  the  mercy  of  ill-trained  clerical  staff  members  who 
aggregate  the  data  to  begin  with.  They  do  not  want  their  role  to  be  reduced  to  a  passive  one  of 
receiving  information  summaries  and  twice-a-day  briefings.  Rather,  they  want  to  have  an  active, 
information-seeking  role.  Their  hypotheses  about  the  nature  of  the  situation  will  determine  the 
types  of  data  that  are  diagnostic  and  easily  obtained. 

Moreover,  there  is  no  basic  level  of  data,  as  if  data  elements  were  primitives.  The 
relevant  data  are  a  function  of  the  way  the  situation  is  understood.  In  a  study  of  weather 
forecasters,  Pliske,  Klinger,  Hutton,  Crandall,  Knight,  &  Klein  (1997)  found  that  the  most  highly 
skilled  forecasters  would  sometimes  hand-plot  the  data,  rather  than  rely  on  computer-generated 
plots,  because  the  grain  of  the  computer  system  was  too  coarse  at  a  given  locale  to  capture  some 
subtle  features.  The  skilled  forecasters  did  not  want  to  begin  their  workshift  with  a  briefing 
about  the  situation,  or  with  a  computer-generated  report.  They  wanted  to  look  back  over  the  data 
for  the  past  several  hours  and  generate  their  own  mental  model. 

Because  experienced  decision  makers  can  identify  trends  and  subtleties  in  the  data  that 
their  subordinates  cannot  see,  they  often  direct  their  subordinates  not  to  filter  any  data,  but  to 
pass  along  everything.  This,  of  course,  results  in  the  information  explosion.  Skilled  decision 
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makers  do  not  need  to  see  all  the  data  or  all  the  information.  They  need  to  be  able  to  drill  down 
for  the  diagnostic  data  in  a  situation,  based  on  their  understanding.  Thus,  the  nature  of  the 
understanding  defines  the  data  —  the  granularity  and  features  of  the  data  that  are  useful. 

Taken  together,  this  critique  of  the  beliefs  presented  in  Table  1  provides  a  picture  of  the 
importance  of  expertise  in  NDM.  The  point  is  not  just  that  expertise  is  important.  The  point  is 
to  describe  how  it  is  important,  how  it  affects  decision  making,  planning,  situation  awareness, 
and  problem  solving. 


V.  BARRIERS  TO  EXPERTISE 

Just  as  we  have  learned  a  great  deal  about  different  aspects  of  expertise  and  how  they  are 
involved  in  NDM,  we  have  also  learned  a  great  deal  about  ways  to  interfere  with  expertise.  In 
this  section,  we  describe  some  of  the  primary  barriers  that  will  prevent  decision  makers  from 
using  their  intuition.  The  barriers  selected  are  directly  linked  to  information  technology.  That  is, 
stressors  such  as  time  pressure  and  uncertainty  are  not  included  because  they  are  not  necessary 
aspects  of  information  technology. 

In  addition,  the  section  describes  how  these  barriers  are  likely  to  affect  military 
operations  related  to  Information  Dominance.  The  claim  here  is  simple:  attempts  at  Information 
Dominance  run  a  strong  risk  of  reducing,  rather  than  improving  performance,  because  the 
technologies  are  often  at  odds  with  human  expertise.  Only  by  gaining  a  clear  picture  of  expertise 
can  we  take  the  necessary  steps  to  avoid  or  reduce  these  barriers. 

Table  2  lists  six  primary  barriers  to  expertise  that  are  posed  by  information  technology. 
The  six  barriers  listed  in  Table  2  will  be  discussed  in  order,  and  each  subsection  will  indicate  the 
ways  that  Information  Dominance  can  exacerbate  the  difficulty.1 

Table  2.  Barriers  to  Expertise  That  Are  Posed  by  Information  Technology 


•  Excessive  data 

•  Pre-processed  data 

•  Excessive  procedures 

•  Performing  formal  analyses 

•  Passive  data  handling  (reactive  mindset),  limited  ability  for  information  seeking 


1  This  discussion  of  barriers  to  expertise  does  not  include  all  of  the  potential  barriers  to  expertise  within  the 
domain  of  Information  Dominance.  For  purposes  of  this  discussion  I  identified  barriers  that  result  from  our  own 
information  systems.  I  did  not  discuss  barriers  to  expertise  that  result  from  actions  that  an  enemy  could  potentially 
take  to  reduce  the  effectiveness  of  our  information  systems,  such  as  introducing  a  disinformation.  Nor  did  I  address 
barriers  to  the  expertise  involved  in  detecting  opportunities  for  disrupting  the  enemy’s  information  systems 
awareness. 
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Interfaces  that  obscure  the  big  picture 


•  Excessive  Data.  Decision  makers  would  ideally  select  and  use  only  relevant 
information  when  presented  with  a  large  amount  of  information.  However,  a  review 
of  the  relevant  research,  in  both  laboratory  and  applied  settings,  found  that  decision 
makers  frequently  select  irrelevant  data  and  that  use  of  these  irrelevant  data  adversely 
affects  their  decision  making  performance  (Gaeth  &  Shanteau,  1984).  More  recent 
studies  have  continued  to  uphold  these  conclusions.  For  example,  Stewart,  Moninger, 
Heideman,  &  Reagan-Cirincione  (1992)  studied  expert  meteorologists  and  found  that 
as  the  amount  and  quality  of  the  information  increased,  there  was  a  substantial 
decrease  in  agreement  among  the  forecasters  regarding  their  predictions  of 
microbursts.  Similarly,  Lusk  and  Hammond  (1991)  demonstrated  that  forecast 
accuracy  did  not  increase  with  increasing  information. 

In  the  past,  when  it  was  observed  that  a  decision  maker  only  utilized  a  subset 
of  the  available  data,  it  has  been  seen  as  a  shortcoming  of  humans,  an  inability  to 
process  large  amounts  of  data.  We  can  change  this  perspective,  and  appreciate  the 
importance  of  expertise  for  enabling  skilled  decision  makers  to  select  the  small  set  of 
relevant  data  with  which  to  work. 

Information  technologies  are  exciting  because  they  can  present  enormous  amounts  of 
data  to  decision  makers,  at  very  high  rates.  This  sounds  wonderful,  until  we  realize  that  these 
data  rates  will  likely  be  excessive,  and  may  result  in  a  degraded  use  of  intuition  and  experience. 
In  some  ways,  the  very  premise  of  information  technology  runs  counter  to  the  effective  use  of 
experience.  We  must  develop  ways  to  facilitate  decision  makers  so  that  they  can  take  advantage 
of  the  enormous  amounts  of  data  that  they  now  have  access  to,  without  causing  them  to  be 
overwhelmed  by  the  sheer  volume  of  information  available. 

•  Pre-processed  Data.  Experts  prefer  to  build  their  own  mental  models  rather  than  rely 
on  the  aggregation  and  analyses  of  subordinates  who  are  less  skilled.  The  activity  of 
building  a  mental  model  itself  provides  a  feel  for  the  situation.  In  one  project 
(Klinger,  Andriole,  Militello,  Adelman,  Klein,  &  Gomes,  1993),  we  heard  from  old- 
time  AW  ACS  weapons  directors  that  in  previous  positions,  as  ground  controllers, 
they  had  to  watch  radar  screens  and  mark  targets  with  grease  pencils.  And  they  felt 
they  had  a  better  feel  for  the  situation  than  they  did  as  AW  ACS  weapons  directors, 
with  computers  to  automatically  enter  and  tag  the  aircraft.  The  manual  engagement 
with  the  screen  helped  them  to  own  their  big  picture. 

Information  technologies  handle  the  data  explosion  by  relying  on  various 
types  of  fusion  algorithms,  artificial  intelligence,  and  related  procedures.  Skilled 
decision  makers  are  often  forced  to  rely  on  pre-processed  data.  A  critical  issue  that 
must  be  addressed  when  designing  future  information  systems  is  who  needs  access  to 
which  raw  data?  Should  the  general’s  staff  always  pre-process  the  data  before  it  is 
passed  to  him?  Research  from  the  NDM  perspective  suggests  that  limiting  high-level 
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decision  makers  to  pre-processed  data  can  be  a  barrier  to  expertise.  Information 
technologies  must  be  exploited  to  allow  high  level  decision  makers  to  have  access  to 
relevant  subsets  of  raw  data  when  necessary. 

•  Excessive  Procedures.  Experienced  decision  makers  are  flexible  in  how  they  perform 
tasks  and  adapt  to  situations.  This  provides  high  levels  of  efficiency.  It  also  allows 
them  to  work  in  context,  rather  than  mindlessly  performing  routinized  actions  that  do 
not  always  make  sense,  because  no  routines  can  anticipate  the  different  challenges  of 
chaotic  battlefields.  If  we  require  experts  to  follow  routines,  we  are  eliminating  one 
of  their  important  strengths.  Pliske  et  al.  (1997)  found  that  expert  weather  forecasters 
are  extremely  flexible  in  their  information  search  strategies  depending  on  the  nature 
of  the  weather  situation  at  hand  (e.g.,  potential  for  high  winds,  potential  for  hail,  etc.). 
Unfortunately,  the  automated  information  system  that  Air  Weather  Service 
forecasters  are  required  to  use  to  produce  their  forecasts  is  not  flexible;  it  had  been 
designed  to  support  a  highly  proceduralized  forecasting  process.  Although  this  highly 
proceduralized  system  helped  novice  forecasters  produce  mediocre  forecasts,  this 
system  actually  served  as  a  barrier  to  the  true  experts. 

Information  technologies  will  likely  require  highly  routinized  actions  in  order  to  avoid 
system  errors,  and  in  order  to  adapt  to  the  needs  of  the  knowledge  bases  regarding  the  type  of 
data  needed  and  the  format  that  is  necessary.  However,  these  technologies  can  also  be  used  to 
develop  flexible  human-computer  interfaces  that  will  support  decision  makers  of  various  skill 
levels.  Highly  skilled  human  decision  makers  must  be  allowed  to  override  routinized  functions  in 
information  systems  that  interfere  with  their  ability  to  utilize  their  expertise. 

•  Performing  Formal  Analyses.  The  act  of  making  factors  explicit,  which  is  central  to 
decision  analysis,  may  itself  interfere  with  subtle  judgments.  A  number  of  studies 
have  reported  this  fairly  surprising  finding  (e.g.,  Schooler  &  Wilson,  1991;  Erev, 
Bomstein,  &  Wallsten,  1993;  Reber,  1993),  that  if  you  ask  decision  makers  to 
produce  a  judgment  for  a  complex  task,  the  judgments  are  reasonably  good.  But  if 
you  first  require  the  decision  makers  to  perform  formal  analyses  of  the  elements  of 
the  situation,  and  then  ask  them  to  make  an  overall  judgment,  the  quality  of  those 
judgments  is  significantly  reduced.  Therefore,  we  can  interfere  with  intuition  by  first 
having  the  decision  makers  perform  analysis. 

The  fascination  with  hard  data  (i.e.,  quantitative  data)  is  in  part  because  of  its 
objectivity,  in  part  because  of  the  ease  of  performing  sophisticated  analyses. 

However,  when  hard  data  are  uncoupled  with  the  soft  data  that  are  needed  to  interpret 
them  and  put  them  in  context,  the  results  can  be  very  misleading. 

Information  technologies  will  often  require  very  formalized  inputs  in  order  to  run  the 
decision  support  systems,  and  these  inputs  are  non-intuitive.  They  rely  on  formal  analyses  and 
decompositions,  all  of  which  will  lead  decision  makers  away  from  their  intuitions.  Information 
technologies  need  to  be  exploited  to  support  intuitive  judgments  as  well  as  analytic  judgments. 
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•  Passive  Data  Handling  (Reactive  Mindset!  Limited  Ability  for  Information  Seeking. 
As  discussed  above,  skilled  decision  makers  rely  on  an  active  engagement  with  the 
situation.  They  rely  on  a  powerful  repertoire  of  information-seeking  strategies.  They 
need  to  build  their  own  mental  models.  If  you  prevent  them  from  taking  an  active 
role,  if  you  put  commanders  into  a  passive  role  of  receiving  data,  if  you  put  C2  staff 
into  a  passive  role  of  scrolling  through  emails,  you  reduce  their  ability  to  use  their 
expertise. 

The  neuropsychologist  Oliver  Sachs  (1995)  wrote  about  a  case  in  which  a  50- 
year  old  man  who  had  lost  his  vision  in  early  childhood,  due  to  thick  cataracts,  had 
recently  regained  his  vision  through  a  simple  cataract  removal  operation.  After 
studying  this  man,  Sachs  realized  that  there  was  still  a  major  disability.  The  man  now 
had  fairly  good  acuity.  He  could  see  shapes  and  colors.  However,  during  the  decades 
of  blindness  he  had  lost  the  ability  to  see.  That  is,  he  was  content  to  passively  receive 
a  succession  of  images.  But  he  did  not  naturally  look  at  things,  or  look  for  things. 
Upon  being  introduced  to  people,  he  did  not  really  look  at  their  faces.  “Virgil  would 
look,  would  attend  visually,  only  if  one  asked  him  to  or  pointed  something  out  —  not 
spontaneously.  His  sight  might  be  restored,  but  using  his  eyes,  looking,  it  was  clear, 
was  far  from  natural  to  him;  he  still  had  many  of  the  habits,  the  behaviors,  of  a  blind 
man.”  (p.  117).  Sachs  concluded  that  the  patient  was  mentally  blind. 

If  we  make  this  distinction  between  passive  receipt  of  images  and  active 
looking,  then  we  can  see  how  much  is  lost  when  we  reduce  decision  makers  to  a 
passive  mode. 

Information  technologies  have  a  very  strong  tendency  to  reduce  users  to  passive 
recipients  of  data,  particularly  users  who  are  not  intimately  familiar  with  the  way  the  hardware 
and  software  is  designed,  and  so  are  reluctant  to  try  to  work  around  problems  or  strike  out  on 
their  own.  We  may  be  creating  a  new  breed  of  commanders  who  become  mentally  blind  in  that 
they  will  have  lost  their  ability  to  look,  to  search.  Moreover,  the  Information  Dominance 
architectures  will  be  designed  by  technophiles  who  may  have  little  appreciation  for  the  demands 
of  combat.  Technophiles  often  try  to  design  technically  impressive  systems,  to  impress  their 
colleagues,  and  view  the  users  as  the  potential  weak  link  in  the  cycle.  Technophiles  sometimes 
see  the  operator  as  a  way  to  feed  the  system,  and  strive  to  design  a  system  that  offers  minimal 
opportunities  for  the  operators  to  stray  from  the  intended  strategies.  System  designers  need  to 
keep  in  mind  that  the  human  expert  needs  to  be  actively  engaged  in  the  information-seeking 
functions  of  the  information  system. 

•  Interfaces  That  Obscure  the  Big  Picture.  It  is  easy  to  reduce  functional  expertise  by 
denying  decision  makers  a  chance  to  see  the  big  picture.  This  can  be  done  in  many 
ways,  by  overloading  a  display  with  data,  by  formatting  the  data  inconveniently,  by 
letting  variables  change  without  calling  attention  to  themselves,  by  requiring  an 
annoying  number  of  secondary  tasks  to  update  a  screen,  by  using  confusing  symbols. 
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and  so  forth.  The  field  of  human-computer  interface  has  been  discovering  new 
methods  for  hiding  the  big  picture  for  several  decades. 

In  addition,  simplistic  slogans  often  masquerade  as  approaches.  The  term 
“user-friendly”  quickly  wore  out  its  welcome.  Soon,  the  term  “user-centered  design” 
will  do  the  same.  The  facile  suggestion  to  rely  more  on  graphics  and  less  on 
alphanumerics,  ignores  findings  that  for  some  tasks,  such  as  inducing  patterns  in  data, 
numerical  data  are  more  useful  than  graphical  representations;  the  issue  of  graphical 
versus  alphanumeric  depends  on  the  nature  of  the  task.  In  short,  slogans  are  not  going 
to  substitute  for  careful  studies  of  the  cognitive  requirements  for  the  tasks. 

Information  technologies  could  easily  distort  the  big  picture  because  the  information 
battlefields  they  will  portray  are  even  more  complex  than  the  geographical  battlefields  with 
which  we  are  familiar.  Since  we  never  learned  how  to  do  a  good  job  of presenting  the  big 
picture  for  the  geographical  battlefields,  we  face  a  tremendous  challenge  in  this  area.  If  system 
designers  take  into  consideration  the  cognitive  requirements  of  the  intended  user  of  the 
information  system,  then  information  technologies  could  be  utilized  to  support  the  development 
of  the  big  picture  rather  than  obscure  it. 

The  arguments  presented  in  this  section  can  be  synthesized  into  a  single  claim:  the  quest 
for  Information  Dominance  runs  a  real  risk  of  reducing  rather  than  improving  the  performance 
of  decision  makers.  The  reason  for  this  is  because  information  technologies  can  interfere  in  a 
major  way  with  the  expression  of  expertise  on  the  part  of  skilled  commanders  and  staff  members. 
Section  VIII  will  explore  some  suggestions  for  avoiding  these  barriers.  But  we  can  only  avoid 
them  if  we  understand  what  they  are. 

Up  to  this  point,  the  report  has  concentrated  on  the  individual  decision  maker.  The  next 
sections  change  this  focus,  and  examine  the  team  decision-making  process. 


VI.  REVIEW  OF  THE  LITERATURE  ON  TEAM  MIND  AND  TEAM 

DECISION  MAKING 

For  Information  Dominance,  much  of  the  decision  making  will  take  place  in  the  context 
of  teams.  The  nature  of  expertise  is  different  at  the  team  level  than  it  is  at  the  individual  level. 

In  this  section  of  the  report,  we  examine  expertise  at  the  team  level.  More  specifically,  we  are 
interested  in  the  concept  of  a  “team  mind,”  an  emergent  phenomenon  in  which  the  team 
understands  and  thinks  and  decides  in  ways  that  transcend  the  individuals.  For  example,  an  air 
campaign  planning  process  depends  on  a  team  mind.  One  element  updates  the  target  nomination 
list.  A  second  element  updates  the  available  airfields  and  aircraft.  A  third  element  prioritizes  the 
target  list.  A  fourth  element  assigns  resources  to  targets.  A  fifth  element  prepares  the  Air 
Tasking  Order.  With  specific  instructions,  each  of  these  elements  may  be  able  to  work  at  the 
procedural  level,  and  few  of  the  tasks  require  much  decision  making.  Yet  the  team  as  a  whole  is 
engaging  in  a  complex  planning  and  problem-solving  activity.  No  individual  member  of  the 
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team  may  understand  the  deep  strategic  implications  of  the  eventual  Air  Tasking  Order. 

Our  challenge  is  to  apply  NDM  at  the  level  of  teams  for  purposes  of  understanding 
Information  Dominance.  We  need  to  clarify  the  cognitive  processes  involved,  the  nature  of 
expertise,  and  the  likely  barriers  to  expertise.  In  order  to  explore  the  implications  of  the  NDM 
framework  for  Information  Dominance  in  the  context  of  teams,  it  will  be  useful  to  consider  NDM 
models  of  team  performance.  Many  different  models  of  team  performance  have  been  proposed 
and  we  review  relevant  models  in  Appendix  A.  In  this  section,  we  briefly  describe  some  of  our 
own  work  on  team  decision  making  that  has  relevance  to  the  domain  of  Information  Dominance. 

Advanced  Team  Decision  Making 

Thordsen,  Klein  and  Kyne  (1994)  developed  a  model  of  Advanced  Team  Performance, 
based  on  the  Advanced  Team  Decision  Making  work  by  Zsambok,  Klein,  Kyne,  &  Klinger 
(1993).  Thordsen  et  al.  relied  on  a  cognitive  metaphor,  viewing  teams  as  cognitive  entities  with 
an  identity  and  the  ability  to  think  and  monitor  itself.  The  goal  of  this  work  was  to  develop  a 
theory-based  framework  which  would  enable  observers  to  assess  the  skills  and  deficiencies  of 
teams  as  they  conducted  business  in  either  simulated  or  actual  environments. 

Based  on  observation  of  strategic  decision-making  teams,  Thordsen  et  al.  (1994) 
identified  13  key  behaviors  that  are  essential  to  high  performance  teams.  These  are  organized 
into  the  ATDM  model.  The  four  components  of  the  ATDM  model  are  Team  Competancies, 
Team  Identity,  Team  Cognition,  and  Team  Metacognition.  Each  of  these  components  is 
composed  of  directly  observable  behaviors.  The  components  and  behaviors  are  depicted  in 
Figure  6. 

Team  Competencies  refer  to  the  skills  of  the  individual  team  members,  and  the  degree  to 
which  the  team  has  mastered  common  routines  for  action.  Team  Identity  describes  the  extent  to 
which  team  members  conceive  of  the  team  as  an  interdependent  unit,  and  then  operate  from  that 
perspective  while  engaged  in  their  tasks.  In  other  words,  are  the  team  members  able  to  decenter 
from  their  individual  roles  to  consider  the  team  as  a  whole?  Team  Cognition  captures  the  notion 
of  a  team  as  an  intelligent  entity,  a  “team  mind”  that  thinks,  solves  problems,  makes  decisions, 
and  takes  actions  collectively  on  a  level  of  complexity  and  sophistication  that  matches  the 
demands  of  the  task.  Team  Metacognition  is  a  regulatory  process  for  all  other  processes 
described  in  the  model.  Self  monitoring  is  a  metacognitive  process  which  helps  teams  promote 
advanced  team  decision  making,  moving  from  weak  to  strong  identity  and  from  a  low  to  high 
conceptual  level  by  determining  how  successfully  the  team  is  using  key  behaviors  and  making 
necessary  adjustments.  Team  Metacognition  by  definition  is  the  ability  of  a  team  to  observe 
itself  while  acting  to  accomplish  its  tasks. 


How  good  are  the 
team  members? 


Team  Competencies 

Are  they  still  struggling  . 

with  basic  procedures?  J 


Is  anyone 
"out  of  it"? 


/^Does  everyone  know 
who  does  what? 

Team  Identity 

Do  people  help_  ls  anyone 

micromanaging?/ 


each  other  out? 


Who’s  taking 
responsib  ility? 


Do  they  spot  and^^ 
correct  problems? 

a 


Team  Metacognition 

Do  they  get  crunched 

V _  for  time? _ 


^re  they  headed  for  Does  everyone  hav^\ 
the  same  goals?  the  same  picture? 

Team  Cognition 

Are  they  behind  the  Do  they  get  paralyzed! 
power  curve?  by  uncertainty? 


Figure  6:  Advanced  Team  Decision-Making  model. 

For  the  component  of  Team  Competencies,  there  are  two  behavioral  markers,  the  skills  of 
the  team  members  and  the  mastery  of  routines  by  the  entire  team.  For  the  component  of  Team 
Identity  there  are  four  dimensions  or  behavioral  markers.  Defining  roles  and  functions  (“Does 
everyone  know  who  does  what?”)  concerns  the  extent  to  which  teams  ensure  that  all  their 
members  know  what  they  are  expected  to  do  to  attain  the  team  goals.  Engaging  (“Is  anyone  ‘out 
of  it?”’)  concerns  the  extent  to  which  team  members  are  involved  in  the  team  task  and  their  own 
functions.  It  also  concerns  encouraging  other  members  to  engage  in  the  team  task. 

Compensating  concerns  the  ability  of  team  members  to  step  outside  their  assigned  roles  or 
functions  and  perform  different  ones  in  order  to  help  the  team  reach  its  goals.  This  dimension 
includes  compensating  when  problems  arise  as  well  as  trying  to  leam  what  caused  the  problem. 
Avoiding  micro  management  is  seen  as  key  to  advanced  team  performance.  Team  members  take 
steps  to  manage  information,  tasks,  or  people  at  an  appropriate  level  of  detail. 

The  component  of  Team  Cognition  also  contains  four  behavioral  markers.  Envisioning 
goals  and  plans  concerns  the  ability  of  teams  to  use  specific,  concrete  language  that  is  put  into  a 
context  relevant  to  the  team  members,  both  through  examples  that  relate  to  their  experience  and 
through  outcomes  that  contrast  success  and  failure.  The  time  horizon  (“Are  they  behind  the 
power  curve?”  refers  to  where  attention  is  directed.  Managing  uncertainty  concerns  the  ability  of 
a  team  to  discover  and  fill  holes  in  the  team’s  information  base  and  to  recognize  and  handle 
inconsistencies  or  contradictions  that  might  be  present.  Achieving  shared  situation  awareness 
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involves  actively  seeking  a  variety  of  views  from  team  members  about  plausible  situation 
assessments,  and  then  ensuring  that  all  members  share  a  common  understanding  of  the 
assessment  that  the  team  eventually  accepts. 

The  component  of  Team  Metacognition  concerns  three  key  behaviors.  Self-monitoring 
(“Do  they  spot  and  correct  problems?”)  is  the  team’s  ability  to  modify  the  way  it  is  performing 
when  problems  are  discovered  through  the  monitoring  function.  Time  management  refers  to  the 
ability  of  a  team  to  meet  goals  before  deadlines  overtake  them,  to  sequence  subtasks  effectively 
so  that  output  from  one  task  connects  where  and  when  it  should  as  input  to  the  next  task,  and  to 
maintain  a  strategic  allocation  of  time  to  subtasks  in  terms  of  their  priority  for  reaching  the 
overall  goal.  Leadership  (“Who’s  taking  responsibility?”)  refers  to  the  ability  of  the  team  to 
perform  leadership  functions,  either  using  the  designated  leader  or  through  workarounds  where 
necessary. 

Klein  and  Thordsen  (1989)  have  introduced  the  term  “team  mind”  to  refer  to  the 
emergent  cognition  at  the  team  level  in  settings  such  as  cockpits.  We  described  different  levels 
of  awareness  for  information  items.  If  all  the  individuals  in  an  airplane  cockpit  know  something, 
it  is  at  the  level  of  the  collective  consciousness.  If  only  one  member  knows  it,  and  does  not  share 
it  with  others,  we  can  say  that  the  knowledge  is  at  the  pre-conscious  level.  We  went  on  to 
describe  cognitive  processes  that  can  be  adapted  to  teams:  the  concept  of  reaction  time,  of 
working  memory  (a  team  can  only  discuss  one  thing  at  a  time),  of  long-term  memory,  situation 
awareness,  and  inferences.  One  of  the  difficulties  teams  will  have  is  to  derive  inferences  when 
the  elements  are  held  at  the  pre-conscious  level  —  that  is,  if  individual  (a)  knows  one  thing,  and 
individual  (b)  knows  another,  the  fusion  of  those  aspects  of  knowledge  into  a  higher  level 
inference  can  become  difficult.  Wegner  (1987)  has  amplified  the  concept  of  a  team’s  long-term 
memory. 

Klein  and  Miller  (1997)  have  studied  distributed  planning  teams  in  a  variety  of  domains 
(Joint  Force  Air  Component  Commander  [JFACC]  planning,  U.S.  Marine  Corps  regimental 
command  post  planning,  Patriot  missile  battery  planning,  air  campaign  planning)  and  have 
developed  a  model,  shown  in  Figure  7,  that  includes  the  planning  functions  themselves  (e.g., 
detecting  problems,  generating  courses  of  action,  evaluating  the  plans),  the  types  of  plans 
required  (e.g.,  modular  versus  integrated  plans,  conceptual  versus  detailed  plans),  and  the  forcing 
functions  in  the  environment  that  determine  the  process  and  product  (e.g.,  time  pressure,  resource 
limitations).  Using  this  model,  we  were  able  to  note  that  in  JFACC  planning,  the  intent  for 
striking  targets,  that  was  part  of  the  target  nomination  cell’s  deliberations,  was  not  adequately 
captured  when  the  planning  package  was  sent  to  the  combat  operations  cell,  with  the  result  that 
targets  were  sometimes  hit  with  inappropriate  munitions.  In  addition,  the  modular  types  of  plans 
(air  tasking  orders)  that  were  produced  were  not  suitable  for  overall  evaluation,  and  as  a  result, 
did  not  get  evaluated.  These  functions  were  simply  not  relevant  for  the  JFACC  plans,  whereas 
they  were  extremely  relevant  for  Patriot  missile  batteries,  where  the  intent  about  which  targets 
needed  the  most  protection  was  clear,  and  the  evaluation  of  the  plans  (how  the  orientation  of  the 
batteries  provided  double  or  triple  coverage)  was  carefully  accomplished. 
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Figure  7:  Distributed  planning  teams. 

Thordsen,  McCloskey,  Heaton,  and  Serfaty  (1996)  have  studied  power  projection  in  naval 
air  planning  involving  FA- 18,  EA6B,  and  E2C  aircraft.  Typically,  there  are  few  opportunities 
for  these  pilots  to  prepare  and  rehearse  their  missions  together.  As  a  result,  critical  aspects  of 
coordination  may  not  be  anticipated,  resulting  in  inefficiencies  and  occasionally  inadequate 
performance.  For  example,  the  crew  aboard  the  EA6B  may  not  be  directly  involved  in  certain 
aspects  of  the  strike  mission,  but  they  are  still  listening  in  on  their  radios  to  gauge  how  the 
mission  is  going.  If  critical  messages  are  passed  while  the  EA6B  is  in  a  jamming  mode,  the 
EA6B  team  will  not  hear  the  radio  traffic,  and  will  suffer  a  degraded  situation  awareness. 

Klein,  Wolf  and  Serfaty  (1997)  have  developed  a  model  of  information  management 
which  decomposes  the  primary  functions  of  information  management  (collecting  data,  assigning 
meaning  to  the  data,  and  either  applying  the  inferences,  transmitting  the  information,  or  seeking 
additional  information.  Each  of  these  functions  is  associated  with  typical  errors  which  result  in 
degradation  or  breakdown  of  the  information  management  function. 

This  brief  review  of  the  research  we  have  conducted  on  team  decision  making  indicates 
that  the  nature  of  expertise  is  different  at  the  team  level  as  compared  to  the  individual  level.  In 
the  next  section,  we  discuss  the  role  of  expertise  in  team  decision  making  in  more  detail. 
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VII.  THE  ROLE  OF  EXPERTISE  IN  TEAM  DECISION  MAKING 


The  interaction  between  naturalistic  decision  making  work  and  teams  is  fairly  subtle,  and 
is  best  understood  by  looking  at  what  is  not  happening,  rather  than  at  what  is  being  addressed. 
Traditional  decision  research  emphasizes  a  Rational  Choice  model  for  individuals,  and  therefore 
would  direct  attention  to  the  use  of  a  Rational  Choice  model  (i.e.,  multi-attribute  utility  analysis) 
for  teams.  Under  a  traditional  framework,  we  should  be  trying  to  help  teams  by  finding  ways  for 
them  to  work  together  to  generate  more  options  to  be  considered,  by  helping  them  identify  more 
dimensions  for  evaluating  these  options,  by  assisting  them  to  properly  weight  the  evaluation 
dimensions,  and  by  supporting  them  in  rating  the  options  and  synthesizing  the  ratings  across 
team  members.  Traditional  decision  researchers  have  studied  ways  of  providing  all  of  these 
types  of  aid,  including  brainstorming  (to  increase  the  set  of  options  and  evaluation  dimensions), 
and  other  techniques  for  combining  the  judgments  and  ratings  of  individual  team  members.  In 
contrast,  NDM  models  do  not  look  at  any  of  these  types  of  methods. 

The  NDM  view  of  team  decision  making  directly  parallels  the  view  of  individual  decision 
making  found  in  accounts  such  as  the  Recognition-Primed  Decision  model  (Klein,  1997b). 
Situation  awareness  is  a  critical  function  for  teams,  and  the  models  of  teamwork  emphasize 
ways  to  use  the  inputs  of  team  members  to  build  a  better  situation  awareness,  and  ways  to 
encourage  shared  situation  awareness  among  the  team  members  (e.g.,  Stout,  Cannon-Bowers,  & 
Salas,  1996).  The  use  of  situation  awareness  to  focus  attention  on  critical  cues  is  important  for 
team  members,  and  one  of  the  challenges  is  to  find  ways  to  selectively  share  information  so  that 
information  overload  is  avoided.  Expectancies  are  important  for  coordinating  team  activities, 
and  for  enabling  all  team  members  to  notice  discrepancies  and  anomalies  and  thereby  to  notify 
the  team  that  the  shared  situation  awareness  may  be  inaccurate.  Situation  awareness  includes  the 
description  of  goals,  and  one  of  the  important  functions  within  team  decision  making  is  for  the 
leader  to  communicate  intent  so  that  the  other  members  can  make  their  own  decisions  about  how 
to  carry  out  directions  and  how  to  improvise.  Situation  awareness  should  provide  a  sense  of 
appropriate  actions,  and  NDM  researchers  are  finding  that  in  many  settings  it  does  not  make 
sense  to  generate  alternative  courses  of  action,  despite  doctrinal  recommendations.  Finally, 
mental  simulation  is  an  important  strategy  for  evaluating  courses  of  action  in  teams  as  well  as  in 
individuals.  The  collective  experience  base  of  the  team  can  be  effectively  utilized  by  having  the 
members  review  how  the  course  of  action  is  intended  to  be  carried  out,  so  that  pitfalls  can  be 
spotted  along  with  opportunities  for  improvement.  Viewed  in  this  way,  the  NDM  approach  to 
team  decision  making  is  radically  different  than  traditional  approaches. 

The  NDM  framework  emphasizes  situation  awareness.  Therefore,  one  approach  we  can 
take  is  to  define  Information  Dominance  as  the  struggle  over  situation  awareness.  Several 
observers  have  converged  on  a  three-part  categorization  of  information  warfare:  attack  the 
enemy’s  information  system  (by  physical  destruction,  deceit,  etc.),  defend  our  own  information 
system,  and  enhance  the  use  of  our  information  system  (U.S.  Air  Force  Scientific  Advisory 
Board,  1995;  Widnall  &  Fogelman,  1995;  Arana-Barradas,  1995;  Ely,  1995;  Whitaker  & 
Kuperman,  1996).  Recasting  this,  we  can  view  Information  Dominance  as  the  attempt  to  disrupt 
the  enemy ’s  situation  awareness,  protect  our  own  situation  awareness  from  attack,  and  enhance 
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our  own  situation  awareness. 


From  this  perspective,  it  is  critical  to  learn  how  teams,  particularly  distributed  teams, 
acquire  a  shared  situation  awareness.  These  mechanisms  may  offer  important  opportunities  for 
attack,  defense,  and  enhancement.  Following  Klein  and  Thordsen  (1989),  if  a  team  mind  is 
analogous  to  an  individual  mind,  then,  borrowing  from  the  RPD  model,  we  would  suggest  that  a 
team’s  situation  awareness  should  hinge  around  the  critical  cues  within  the  context  of  the 
mission,  the  expectancies,  the  goals  being  pursued,  and  the  typical  courses  of  action,  or  the  most 
promising  leverage  points  and  most  threatening  choke  points. 

Because  information  seeking  is  so  central  to  building  situation  awareness,  we  may  want 
to  pay  particular  attention  to  the  way  teams  go  about  their  information  seeking.  One  common 
strategy  is  to  announce  critical  information  requirements.  We  have  observed  many  exercises 
where  this  was  done.  In  almost  all  cases,  these  critical  information  requirements  were  ignored 
throughout  the  entirety  of  the  exercise.  There  is  a  great  deal  of  work  that  needs  to  be  done  here. 


VIII.  BARRIERS  TO  EXPERTISE:  THE  IMPACT  OF  INFORMATION 
DOMINANCE  ON  TEAM  DECISION  MAKING 

In  Section  V  we  covered  a  set  of  barriers  to  proficiency  at  the  individual  level  of  NDM. 
We  identified  a  set  of  conditions  we  could  impose  that  would  diminish  expertise,  and  we  showed 
how  each  of  these  was  likely  to  occur  with  the  use  of  information  technologies  for  Information 
Dominance.  In  this  section,  we  will  do  the  same  at  the  team  level.  We  will  describe  how  each  of 
these  barriers  affects  team  decision  making,  using  the  Advanced  Team  Decision  Making 
(ATDM)  model  (shown  again  here  as  Figure  8)  as  our  frame  of  reference. 

The  list  of  barriers  to  expertise  is  the  same  one  that  was  used  in  Section  V: 

•  excessive  data 

•  pre-processed  data 

•  excessive  procedures 

•  performing  formal  analyses 

•  passive  data  handling  (reactive  mindset),  limited  ability  for  information  seeking 

•  interfaces  that  obscure  the  big  picture 
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How  good  are  the 
team  members? 


a 


Does  everyone  know 
who  does  what? 


Is  anyone 
"out  of  it"? 


Team  Competencies 


Are  they  still  struggling 
with  basic  procedures? 


Team  Identity 

Do  people  help^  Is  anyone 

micromanaging? 


each  other  out? 


Who’s  taking 
responsib  ility? 


Do  they  spot  and 
correct  problem  s? 


Team  Metacog nitiorr 


Do  they  get  crunched 
for  time? 


T 

/hTQ  they  headed  for  Does  everyone  have\ 
I  the  same  goals?  the  same  picture?  | 

Team  Cognition 


Are  they  behind  the 
,  power  curve? 


Do  they  get  paralyzed 
by  uncertainty?  J 


Figure  8:  ATDM  model. 

•  Excessive  Data.  This  will  disturb  a  team’s  decision  making  in  several  ways.  With 
regard  to  Team  Identity,  it  will  blur  roles  and  functions  because  as  information  floods 
a  team  the  boundaries  between  individual  roles  and  functions  become  unclear.  If 
there  are  any  weaknesses  in  the  clarity  of  roles  and  functions,  then  with  information 
overload  the  team  members  consider  the  additional  information  to  be  an  added 
burden,  and  not  part  of  their  jobs.  No  one  may  feel  that  a  particular  piece  of 
information  is  his/her  responsibility.  So  data  will  be  lost,  and  the  team  members 
receiving  the  data  will  be  confused  about  what  their  responsibilities  are. 

With  regard  to  Team  Cognition,  excessive  data  will  shrink  the  time  horizon. 
Consider  the  analogy  to  driving.  On  a  highway  it  is  very  easy  to  look  far  up  the  road 
to  search  for  obstacles  and  indications  regarding  a  route.  In  a  city,  a  driver  has  more 
difficulty  —  there  is  too  much  information,  too  many  different  signs  to  be  scanned. 
As  a  result,  the  car  has  to  be  driven  more  slowly,  because  the  look  ahead  is  reduced. 
Similarly,  the  range  of  factors  considered  by  the  team  becomes  compromised.  Teams 
have  more  difficulty  considering  a  wide  range  of  factors,  in  knowing  which  factors 
are  important,  and  in  understanding  how  any  one  factor  impacts  another. 

Information  Dominance  is  designed  to  increase  the  data  stream.  So  we  may  find 
breakdowns  in  Team  Identity  and  Team  Cognition. 
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•  Pre-processed  Data.  If  we  provide  a  team  with  information  and  knowledge  and 
analyses  that  have  already  been  performed,  perhaps  electronically,  it  can  enhance 
their  operations.  But  it  can  also  lead  to  confusions  about  where  the  data  came  from, 
and  how  accurate  they  are.  In  the  ATDM  model,  this  relates  to  the  function  of  Team 
Competencies,  in  which  one  of  the  difficulties  of  teams  is  gauging  the  competence  of 
other  members.  This  is  particularly  true  in  distributed  teams,  where  the  identities  and 
competence  of  the  other  members  are  unknown.  In  battle  command  exercises,  much 
of  the  gain  of  using  pre-processed  data  is  lost  in  the  frantic  search  to  verify  data 
quality  and  source. 

Team  Cognition  is  also  compromised  in  using  pre-processed  data,  particularly 
the  shared  situation  awareness.  Uncertainties  about  data  source  and  quality  will 
typically  result  in  differing  levels  of  confidence  with  the  picture  that  is  painted, 
thereby  interfering  with  the  goal  of  building  a  shared  picture. 

Information  Dominance  will  require  an  increased  level  of  pre-processed  data,  which  may 
create  confusions  about  data  source  and  quality,  and  interfere  with  the  development  of  a  shared 
situation  awareness. 

•  Excessive  Procedures.  By  requiring  teams  to  follow  rigid  procedures,  we  reduce  the 
ability  of  the  teams  to  be  flexible  and  to  adapt  to  conditions.  We  reduce  their 
efficiency.  The  impact  is  seen  most  clearly  in  the  area  of  Team  Metacognition  —  the 
leadership  ability  of  seeing  what  is  working  and  what  isn’t,  and  making  corrections. 

This  is  particularly  visible  in  contexts  such  as  the  JFACC  planning/execution 
cycle.  The  72-hour  cycle  is  rigidly  laid  out.  With  the  deliberateness  of  a  production 
line,  the  Air  Tasking  Order  marches  through  the  various  stations,  from  the  target 
nomination  board  to  the  combat  operations  cells,  and  out  to  the  aviators.  One  of  the 
striking  aspects  of  this  process,  as  we  observed  it,  is  the  minimal  input  of  the 
commander.  In  conventional,  ground-based  combat,  we  understood  that  the  focus  of 
activity  was  not  against  the  enemy  strengths,  not  to  take  advantage  of  the  enemy 
weaknesses,  not  against  the  enemy  troops,  but  against  the  mind  of  the  enemy 
commander.  By  striking  at  the  enemy  commander’s  mind,  he  could  be  persuaded  to 
draw  back  rather  than  strike  boldly,  to  become  reactive  rather  than  proactive.  In  some 
cases,  commanders  have  surrendered  despite  having  superior  troop  strength  and 
resources  compared  to  their  opponents.  And  the  will  of  one’s  own  commander  was 
critical  for  providing  a  vision  of  what  was  to  be  accomplished. 

But  in  the  JFACC  planning  cycle,  we  were  hard  pressed  to  find  evidence  of 
the  impact  of  the  Joint  Force  Air  Campaign  Commander.  The  Air  Tasking  Order  was 
assembled  by  procedures.  The  rationale  was  not  bundled  with  the  targets.  In  fact,  for 
exercises  we  have  observed,  the  only  activity  of  the  commander  was  to  indicate  what 
percentage  of  sorties  were  to  be  allocated  for  interdiction,  close  air  support,  and  so 
forth,  as  the  battle  changed.  We  recognize  that  in  combat  the  role  and  influence  of  the 
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commander  varies  greatly.  This  role,  issuing  percentages,  seems  fairly  trivial;  in  most 
cases,  the  percentages  are  obvious,  and  this  is  a  far  cry  from  generating  a  vision  of 
what  is  to  be  accomplished. 

It  also  struck  us  that  no  one  had  a  particularly  clear  understanding  of  the  big 
picture,  because  there  was  no  grand  plan,  but  rather  a  collection  of  micro-plans  for 
conducting  strikes  against  a  variety  of  targets.  The  expertise  of  the  team  was  in 
carrying  out  procedures,  not  in  gauging  the  effectiveness  of  their  approach,  or 
reconfiguring  their  approach. 

Information  Dominance  may  result  in  a  fundamental  change  in  the  nature  of  combat.  It 
may  eliminate  the  attempt  to  attack  the  mind  of  the  enemy  commander,  and  it  may  render 
obsolete  the  leadership  of  one ’s  own  commander.  It  may  move  us  to  a  proceduralized  planning 
and  implementation  cycle  that  is  difficult  to  adjust  on  the  fly  because  no  one  has  the  big  picture, 
no  one  understands  how  the  process  is  working,  and  no  one  has  the  confidence  to  make  changes 
because  of  a  fear  of  unintended  consequences. 

•  Performing  Formal  Analyses.  Mintzberg  (1994)  has  documented  in  great  detail  the 
ways  that  strategic  planning  has  been  ineffective  and  has  resulted  in  worse 
performance,  rather  than  better  performance.  He  discusses  the  dangers  of  formal 
analyses  performed  by  planning  teams  that  are  disconnected  from  operational 
realities,  attempting  to  decompose  situations  into  elements,  and  relying  on 
quantitative  data  rather  than  on  subjective  impressions.  The  team  decision  making 
function  most  severely  compromised  by  formal  analysis  is  Team  Cognition,  because 
the  quality  of  the  plans,  solutions,  and  decisions  is  degraded.  But  Team  Identity  is 
also  affected  because  the  formal  analyses  are  usually  performed  in  a  way  that 
disengages  the  operational  staff  from  the  planning  staff. 

Information  Dominance  will  call  for  an  increase  in  formal  analyses  because  the 
complexity  of  the  problems  and  the  needs  to  rely  on  intelligent  decision  support  systems  will 
require  quantitative  data  for  decomposed  elements.  The  result  may  be  less  flexible  and 
imaginative  decision  making. 

•  Passive  Data  Handling  fReactive  Mindsef).  Limited  Ability  for  Information  Seeking. 
One  of  the  challenges  at  a  team  level  is  to  know  how  to  exchange  information,  to 
produce  an  effective  shared  situation  awareness.  If  too  many  data  items  are 
exchanged,  the  result  is  an  overload.  And  if  too  few  are  exchanged,  critical  items  will 
not  be  acted  upon. 

A  passive  mentality  allows  both  to  happen.  It  is  easy  to  simply  reroute  emails 
to  everyone,  or  to  generic  user  groups,  knowing  that  there  are  no  consequences  for 
sending  irrelevant  messages.  Thus,  in  Desert  Storm,  General  Boomer  received  1.3 
million  emails  in  the  first  30  hours  of  the  ground  war.  (This  did  not  bother  him 
because  he  spent  almost  no  time  in  his  command  post  during  this  period.) 
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It  is  also  easy  to  exchange  too  few  data  items.  If  there  is  an  important  item  of 
information,  the  holder  needs  to  understand  its  implications  in  order  to  deal  with  it 
appropriately.  But  what  happens  if  the  critical  pattern  is  dispersed  among  several 
individuals?  Each  one  may  discount  the  data  item  he  or  she  is  holding,  because  by 
itself  it  may  have  no  significance.  The  significance  is  only  in  concert  with  the  other 
data  elements.  An  active  mindset  is  needed  to  put  these  pieces  together.  What  will 
happen  in  combat  when  we  must  depend  on  the  individual  team  members  to  actively 
wonder  about  the  deeper  implications  of  certain  data  elements,  and  actively  seek  out 
others  to  check  on  what  is  happening?  A  passive  mindset  will  make  such  behavior 
less  likely. 

In  another  example,  during  a  recent  exercise  one  of  the  officers  noted  that  the 
command  post  had  not  received  any  pilot  reports  for  a  while.  He  wondered  about  this 
(it  is  very  hard  to  detect  negative  cues  —  the  absence  of  events),  checked  with  others, 
and  found  that  there  were  no  pilot  reports  for  the  previous  24  hours.  Further 
investigation  showed  that  someone  in  the  command  post  had  changed  the  makeup  of 
newsgroups  (which  had  been  set  up  to  reduce  the  confusion  over  all  the  email),  and  as 
a  result,  the  pilot  reports  had  been  dropped!  And  no  one  had  noticed,  not  any  of  the 
analysts  sitting  in  front  of  their  screens  reading  their  emails,  except  one  officer  who 
had  maintained  his  active  mindset. 

The  real  challenge  to  a  team  is  to  provide  a  partially  shared  situation 
awareness,  in  which  people  learn  what  they  need  to,  and  are  not  bombarded  with 
excessive  messages.  This  is  difficult  to  work  out,  possibly  impossible.  The  solution 
is  not  to  work  this  out,  but  to  maintain  active  mindsets  so  that  the  team  members  are 
engaged  in  information  seeking,  rather  than  information  receipt. 

Information  technologies  have  a  strong  tendency  to  produce  a  passive  mindset  in  the 
team  members,  and  the  development  of  Information  Dominance  is  likely  to  make  this  worse  by 
increasing  message  flow  and  increasing  the  fear  of  missing  something. 

•  Interfaces  That  Obscure  the  Big  Picture.  We  know  much  less  about  how  to  interfere 
with  the  expertise  of  teams,  than  we  do  about  interfering  with  the  expertise  of 
individual  decision  makers.  Nevertheless,  it  should  be  possible  to  speculate  about 
some  easy  ways  to  disrupt  teams,  using  the  ATDM  model  as  a  starting  point.  We 
would  diminish  the  sense  of  Team  Competencies  by  masking  the  identity  of  the  other 
team  members,  working  at  separate  stations,  and  keeping  the  inputs  of  all  team 
members  stylized  so  that  there  is  little  opportunity  to  gauge  competence.  We  would 
diminish  the  sense  of  Team  Identity  by  making  it  difficult  to  track  who  the  others  are, 
what  parts  of  the  task  they  are  doing,  what  their  progress  is,  where  they  might  need 
help,  and  so  forth.  We  would  diminish  the  sense  of  Team  Cognition  by  preventing  a 
common,  shared  picture  of  the  current  situation,  by  allowing  different  perspectives  on 
the  nature  of  progress,  by  allowing  different  understanding  of  the  goals  of  the 
mission.  And  we  would  diminish  the  sense  of  Team  Metacognition  by  making  it 
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difficult  to  gauge  how  each  of  the  team  members  was  proceeding  (rate  of  progress). 

An  example  of  an  interface  that  was  effective  in  obscuring  the  big  picture  for 
teams  was  recently  observed  in  a  military  exercise  that  was,  ironically,  trying  to  show 
how  information  technology  could  be  valuable  in  presenting  the  same  big  picture  to 
all  the  staff  members  in  the  command  post.  What  happened  was  that  the  large  screen 
display  showed  clusters  of  red  symbology,  indicating  the  presence  of  enemy  units,  but 
upon  closer  inspection  these  were  not  credible.  They  included  elements  such  as 
reported  sightings  of  enemy  tank  units  that  were  24  hours  old  (and  therefore 
obsolete),  but  had  not  yet  been  removed  from  the  screen.  The  display  also  showed 
areas  that  were  clear,  suggesting  that  these  were  safe  zones  for  inserting  troops,  but 
upon  closer  inspection  these  were  not  credible  either.  They  included  areas  where 
there  were  no  sensors,  so  it  was  entirely  possible  that  large  enemy  units  might  be 
there.  Because  of  confoundings  such  as  this,  the  large  screen  display  generated  too 
much  confusion,  and  on  occasion  led  the  officers  in  charge  to  erroneous 
interpretations  of  the  situation. 

As  we  pursue  the  goal  of  Information  Dominance,  the  interface  will  become  an  even  more 
critical  aspect  of  effective  team  decision  making.  The  present  state  of  the  art  is  inadequate  for 
supporting  team  decision  making.  Possibly,  there  are  technological  solutions  just  beyond  the 
current  state  of  the  art,  that  will  solve  these  problems.  More  likely,  they  will  be  only  partial 
solutions,  and  will  create  additional  problems.  The  inadequacy  of  interfaces  may  continue  to 
disrupt  team  decision  making. 

Just  as  we  observed  that  the  quest  for  Information  Dominance  reduced  the  expertise  of 
individual  decision  makers,  so  we  must  also  prepare  for  this  quest  to  reduce  the  expertise  of 
decision-making  teams. 


IX.  NATURALISTIC  DECISION  MAKING  AND  INFORMATION 
DOMINANCE:  RECOMMENDATIONS  AND  RESEARCH  AGENDA 

In  this  report,  I  have  explored  the  implications  of  the  NDM  framework  for  the  domain  of 
Information  Dominance.  I  have  argued  that  we  must  work  to  shape  the  technologies  going  into 
Information  Dominance,  so  that  they  can  support  individual  and  team  decision-making  expertise, 
otherwise  the  technologies  may  actually  hinder  expert  performance.  The  objective  of  developing 
system  requirements  in  service  of  individual  and  team  expertise  is  challenging;  it  has  many  facets 
and  many  complexities. 

Although  this  report  has  raised  a  number  of  cautionary  notes  about  information 
technology,  the  intent  has  not  been  to  criticize  this  technology  or  discourage  its  use.  The  benefits 
of  information  technology  are  clear  enough,  and  have  been  described  in  countless  briefings  and 
articles.  The  argument  presented  in  this  report  is  that  there  are  few  unmixed  blessings,  and  if  we 
do  not  try  to  anticipate  some  of  the  undesirable  consequences  of  information  technology,  we  will 
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be  doing  ourselves  a  disservice.  Our  challenge  is  to  learn  how  to  manage  information  technology 
so  that  it  does  not  compromise  the  expertise  of  individuals  and  teams.  We  need  to  learn  how  to 
shape  information  technologies  to  support  different  aspects  of  expertise,  such  as  making  fine 
discriminations,  anticipating  events,  seeking  diagnostic  data,  detecting  early  signs  of  problems, 
seeing  the  big  picture,  and  so  forth. 

The  NDM  perspective  appears  to  offer  a  number  of  important  directions  for  future  study, 
in  order  to  support  highly  skilled  individual  and  team  decision  making  in  the  Information 
Dominance  arena.  In  this  section  we  list  six  areas.  The  NDM  approach  emphasizes  the 
importance  of  studying  the  decision  maker  within  his  or  her  natural  work  environment.  For  the 
following  recommendations  we  focus  on  the  intelligence  analyst’s  work  environment  for  the  sake 
of  clarity,  and  are  not  suggesting  that  this  is  the  only  decision  maker  relevant  to  the  domain  of 
Information  Dominance. 

First,  it  would  be  highly  instructive  to  conduct  a  Cognitive  Task  Analysis  to  document 
the  cognitive  aspects  of  the  job  of  an  intelligence  analyst.  A  Cognitive  Task  Analysis  would 
document  the  critical  cognitive  demands  that  are  central  to  the  analysts’  job.  It  would  also 
document  the  extent  to  which  current  information  systems  support  these  demands,  and  the  ways 
that  they  degrade  and  interfere  with  them.  Based  on  the  results  of  the  Cognitive  Task  Analysis, 
we  could  develop  a  tangible  database  of  lessons  learned  that  would  be  used  to  guide  future 
system  design  efforts  and  the  other  research  areas  described  below. 

Second,  the  barriers  analyses  presented  in  this  report  for  individual  and  team  decision 
making  should  be  developed  into  a  model  of  the  critical  factors  needed  to  sustain  expertise  for 
intelligence  analysts.  This  model  would  generate  predictions  of  how  well  specific  information 
systems  will  support  (or  degrade)  intelligence  analysts’  performance.  These  predictions  could 
then  be  empirically  validated  using  performance  data  from  analysts  using  existing  systems,  as 
well  as  analysts  using  new  systems  as  they  are  implemented.  The  outcomes  of  the  empirical  tests 
could  then  be  used  to  revise  the  model  in  order  to  improve  its  accuracy  for  use  in  the 
development  of  future  information  systems. 

Third,  research  needs  to  be  conducted  to  identify  how  intelligence  analysts  spot  leverage 
points  that  could  be  used  to  degrade  an  adversary’s  information  systems.  Research  could  also 
identify  the  strategies  that  analysts  use  to  spot  vulnerabilities  in  our  own  systems.  Expertise  in 
Information  Dominance  will  center  around  the  ability  to  see  and  respond  to  these  leverage  points, 
and  so  it  would  be  valuable  to  study  how  skilled  information  warriors  notice  them  in  the  first 
place.  This  research  would  build  on  the  results  of  the  Cognitive  Task  Analysis  and  focus 
specifically  on  how  intelligence  analysts  understand  the  types  of  opportunities  that  allow  us  to 
degrade  an  adversary’s  situation  awareness. 

Fourth,  research  needs  be  conducted  to  discover  new  human-computer  interface  features 
that  support  key  requirements  for  individual  analysts.  The  emphasis  here  is  on  generating  new 
practices,  new  ideas  for  system  and  interface  design.  The  strategy  would  be  to  identify  common 
requirements  for  expertise,  to  document  failures  in  supporting  these  requirements,  along  with 
some  of  the  reasons  for  these  failures,  and  to  explore  strategies  for  providing  effective  support. 
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This  research  would  again  build  on  the  results  of  the  Cognitive  Task  Analysis,  but  would  extend 
that  research  to  address  human-computer  interface  design  issues  for  individual  analysts. 

Fifth,  research  needs  to  be  conducted  to  clarify  the  indicators  of  team  competence  in 
Information  Dominance  by  conducting  a  Cognitive  Task  Analysis  of  distributed  teams  that 
include  individual  intelligence  analysts.  In  order  to  complete  this  research  objective,  new 
methods  of  Team  Cognitive  Task  Analysis  will  need  to  be  developed.  By  developing 
appropriate  methods,  we  will  be  able  to  collect  data  that  will  allow  us  to  compare  different 
information  system  and  interface  designs.  This  research  would  improve  the  performance  of 
individual  intelligence  analysts  by  supporting  the  cognitive  demands  faced  by  the  distributed 
team. 


Sixth,  additional  research  is  needed  to  characterize  the  interactions  in  distributed 
Information  Dominance  teams  in  order  to  study  ways  to  re-engineer  them.  As  information 
technology  continues  to  be  delivered,  there  will  be  ongoing  pressure  to  change  the  composition 
and  configuration  of  Information  Dominance  teams.  Platforms  need  to  be  developed  for 
representing  and  comparing  different  approaches  to  organizing  and  staffing  these  teams.  This 
research  would  extend  the  Team  Cognitive  Task  Analysis  methods  to  explore  the  utility  of 
various  representation  platforms. 

In  conclusion,  NDM  is  the  study  of  how  people  use  their  experience  to  make  decisions  in 
field  settings.  This  report  examined  the  domain  of  Information  Dominance.  The  expertise  we 
considered  was  at  the  individual  and  team  level  of  decision  making.  We  defined  what  expertise 
consists  of,  and  we  identified  some  important  barriers  that  might  be  particularly  troublesome  in 
an  era  of  Information  Dominance.  We  have  also  seen  that  the  NDM  perspective  can  offer  some 
ideas  about  using  information  technology  to  support  skilled  decision  making.  We  have  identified 
several  possible  directions  for  future  research  into  the  barriers  themselves,  the  possibilities  for 
support  of  decision  making,  the  implementation  of  Information  Dominance  to  expand  situation 
awareness  and  decrease  the  adversary’s  situation  awareness,  and  for  enabling  a  battle  staff  to  re¬ 
engineer  its  own  functionality. 
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GLOSSARY 


AI 

Artificial  Intelligence 

ATDM 

Advanced  Team  Decision  Making 

JFACC 

Joint  Force  Air  Component  Commander 

NDM 

Naturalistic  Decision  Making 

RPD 

Recognition-Primed  Decision 
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APPENDIX  A  -  MODELS  OF  TEAM  PERFORMANCE2 

In  this  Appendix,  several  models  of  teamwork  are  reviewed.  These  models  describe  team 
processes.  Specifically,  we  focus  our  attention  on  five  different  models  of  team  performance  that 
have  been  used  to  formulate  assessment  tools  to  evaluate  teams  (Fleishman  &  Zaccaro,  1993; 
McIntyre  &  Dickinson,  1992;  Morgan,  Glickman,  Woodard,  Blaiwes,  &  Salas,  1986;  Olmstead, 
1992;  Helmreich  &  Foushee,  1993).  A  sixth  model  (Zsambok,  Klein,  Kyne,  &  Klinger,  1993) 
was  described  in  the  body  of  this  report.  Although  these  models  use  different  terms,  there  is 
really  a  great  deal  of  similarity  in  the  concepts.  Some  of  the  models  do  not  address  the  decision 
making  function  per  se,  but  all  of  them  are  relevant  to  NDM  because  they  consider  factors  that 
would  have  a  direct  bearing  on  a  team’s  ability  to  make  effective  decisions. 

We  conclude  the  section  with  a  brief  discussion  of  other  accounts  of  team  mind. 

Fleishman  and  Zaccaro  ('1993'):  Team  Performance  Model 


Fleishman  and  Zaccaro  prepared  an  extensive  taxonomy  of  team  functions,  and  derived  a 
model  by  synthesizing  the  behaviors  they  catalogued.  Their  model  has  been  applied  to  military 
domains,  primarily  Navy  teams. 

The  earliest  step  in  the  evolution  of  the  Team  Performance  Model  was  the  work  of  Nieva, 
Fleishman,  and  Rieck  (1978),  who  developed  a  model  of  team  performance  based  upon  an 
extensive  review  of  the  group  performance  literature.  They  concluded  that  team  performance  has 
two  primary  components:  individual  task  behaviors,  or  those  behaviors  requiring  no  coordination 
among  team  members;  and  coordinated  task-related  processes/functions/behaviors,  including  all 
behaviors  that  promote  coordination  among  individuals,  members,  and  subtasks.  Both  of  these 
components  combine  to  determine  the  level  and  nature  of  group  performance.  The  weight  of 
each  component’s  contribution  to  overall  team  performance  varies  according  to  the  nature  of  the 
particular  task  characteristics. 

Four  antecedent  classes  of  variables  that  influence  team  performance  were  identified 
during  the  literature  review  (Nieva  et  al.,  1978).  These  include:  external  conditions  imposed  on 
the  team,  member  resources,  team  characteristics,  and  task  characteristics  and  demands. 

External  Conditions  imposed  on  the  team  refers  to  both  the  larger  organizational  system  in  which 
the  team  is  embedded,  and  the  team’s  physical  environment.  These  external  conditions  often 
determine  the  amount  of  time  available  to  complete  the  task,  job  operating  procedures,  intergroup 
communication  patterns,  and  power  and  authority  distribution.  The  team  must  either  adapt  to, 
model,  or  be  influenced  by  these  changing  elements.  The  changing  environmental  conditions 


2 

Adapted  from  Militello,  Kyne,  Klein,  Getchell-Reiter,  &  Thordsen  (1994).  Comparing  models  of  team 
performance.  Fairborn,  OH:  Klein  Associates  Inc.  Prepared  under  Contract  MDA903-92-C-0098  for  the  U.S.  Army 
Research  Institute,  Alexandria,  Virginia. 
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thus  influence  member  resources,  team  characteristics,  and  task  characteristics  and  demands. 

Member  resources  include  the  knowledge,  skills,  abilities,  and  experiences  that  members 
bring  to  the  team’s  task.  In  addition,  the  members’  motives,  attitudes,  personality  characteristics, 
and  traits  are  considered  performance-related  resources.  These  resources  can  combine  to  form 
relationships  that  enhance  or  degrade  team  performance.  Team  characteristics  are  defined  as 
those  that  apply  to  the  group  as  a  whole  rather  than  to  specific  individuals.  Examples  of  team 
characteristics  include  group  cohesion,  size,  structure,  and  authority  structure.  The  Team 
Performance  Model  specifies  that  team  variables  are  affected  by  both  member  resources  and  task 
characteristics.  Task  characteristics  and  demands  include  the  aspects  of  the  task  that  determine 
process  and  performance  requirements.  These  aspects  of  the  task  determine  the  critical 
requirements  for  successful  performance  and  affect  the  interactions  that  team  members  have  with 
each  other. 

Nieva  et  al.  (1978)  suggest  that  these  antecedents  can  alter  the  nature  of  team  processes. 
In  order  to  fully  understand  how  these  variables  relate  to  team  performance  and  its  underlying 
processes,  the  authors  proposed  a  taxonomy  of  team  functions  which  clearly  specify  particular 
team  performance  functions.  The  taxonomy  in  its  most  recent  form  is  shown  in  Table  3.  It 
contains  seven  major  categories  of  team  performance  functions.  Each  category  of  functions  is 
intended  to  reflect  a  focus  on  task  accomplishment  with  a  concern  for  member 
interconnectedness.  They  are  considered  molar  functions,  cutting  across  specific  member 
activities;  and  the  categories  are  relative,  meaning  that  they  can  be  ordered  with  respect  to  each 
other. 


Orientation  functions  refer  to  the  processes  used  by  team  members  in  the  acquisition  and 
distribution  of  information  necessary  for  task  accomplishment.  This  includes  most  of  the 
activities  in  the  planning  stage  of  team  performance.  Resource  distribution  functions  refer  to 
processes  to  assigning  members  and  their  resources  to  particular  responsibilities.  This  includes 
resource  assignment  across  subtasks.  Timing  functions  (activity  pacing)  are  directed  toward  the 
organization  of  team  resources  and  activities  to  ensure  that  performance  tasks  are  completed 
within  established  temporal  boundaries. 

Response  coordination  functions  refer  to  the  coordination  and  integration  of  independent 
and  synchronized  member  activities.  Motivational  functions  refer  to  the  definition  of  team 
objectives  and  processes  for  motivating  members  to  adhere  to  these  objectives.  This  includes  the 
establishment  and  acceptance  of  performance  norms  and  reward  systems.  Systems  monitoring 
refers  to  the  detection  of  errors  in  the  nature  and  timing  of  ongoing  activities  of  both  the  team  as 
a  whole  and  individual  members.  Procedure  maintenance  refers  to  the  monitoring  of  both 
synchronized  and  individual  actions  to  ensure  compliance  with  established  performance 
standards. 
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Table  3.  Taxonomy  of  Team  Functions:  Current  version.  (Fleishmann  &  Zaccaro, 
1993)  [Reprinted  with  permission] 


Orientation  Functions 

a.  Information  Exchange  Regarding  Member  Resources  and  Constraints 

b.  Information  Exchange  Regarding  Team  Task  and  Goals/Mission 

c.  Information  Exchange  Regarding  Environmental  Characteristics  and  Constraints 

d.  Priority  Assignment  Among  Tasks 

Resource  Distribution  Functions 

a.  Matching  Member  Resources  to  Task  Requirements 

b.  Load  Balancing 

Timing  Functions  (Activity  Pacing) 

a.  General  Activity  Pacing 

b.  Individually-Oriented  Activity  Pacing 

Response  Coordination  Functions 

a.  Response  Sequencing 

b.  Time  and  Position  Coordination  of  Responses 
Motivational  Functions 

a.  Development  of  Team  Performance  Norms 

b.  Generating  Acceptance  of  Team  Performance  Norms 

c.  Establishing  Team-Level  Performance-Rewards  Linkages 

d.  Reinforcement  of  Task  Orientation 

e.  Balancing  Team  Orientation  with  Individual  Competition 

f.  Resolution  of  Performance-Relevant  Conflicts 

Systems  Monitoring  Functions 

a.  General  Activity  Monitoring 

b.  Individual  Activity  Monitoring 

c.  Adjustment  of  Team  and  Member  Activities  in  Response  to  Errors  and  Omissions 
Procedure  Maintenance 

a.  Monitoring  of  General  Procedural-Based  Activities 

b.  Monitoring  of  Individual  Procedural-Based  Activities 

c.  Adjustments  of  Nonstandard  Activities _ _ _ _ _ 
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McIntyre  and  Dickinson  (1992V  Teamwork  Model 


The  Teamwork  Model,  developed  by  McIntyre  and  Dickinson,  was  also  sponsored  by 
NTSC  and  is  based  upon  the  results  of  the  TEAM  studies,  and  a  review  of  the  teamwork 
literature.  The  Teamwork  Model  incorporates  the  findings  of  the  TEAM  studies  that  directly 
address  teamwork  (as  opposed  to  taskwork)  and  those  of  other  researchers  investigating 
teamwork.  This  work  was  carried  out  in  the  context  of  a  Navy  ship’s  combat  information  center, 
where  information  from  a  number  of  sources  must  be  processed  and  acted  upon  by  many 
individuals  before  a  final  decision  can  be  made  to  deal  with  a  threat. 

McIntyre  and  Dickinson  use  a  control  theory  metaphor,  in  that  a  team  is  considered  to 
be  a  communication  network  with  formal,  prescribed  communication,  and  informal,  closed-loop 
communication.  Teamwork  is  considered  to  consist  of  those  behaviors  that  engender  a  sharing  of 
information  and  a  coordination  of  activities:  behaviors  that  are  crucial  to  the  workings  of  the 
Combat  Information  Center  and  many  other  domains  and  tasks.  This  process  of  sharing 
information  and  coordinating  activities  through  teamwork  is  the  focus  of  this  line  of  research. 

Based  upon  several  literature  reviews,  McIntyre  and  Dickinson  identified  seven  core 
components  of  teamwork.  Team  orientation  refers  to  the  attitudes  that  team  members  have 
toward  one  another  and  the  team  task.  It  reflects  acceptance  of  team  norms,  level  of  group 
cohesiveness,  and  importance  of  team  membership.  Team  leadership  involves  providing 
direction,  structure,  and  support  for  other  team  members.  It  does  not  necessarily  refer  to  a  single 
individual  with  formal  authority  over  others.  Team  leadership  can  be  shown  by  several  team 
members.  Communication  involves  the  exchange  of  information  between  two  or  more  team 
members  in  the  prescribed  manner  and  by  using  proper  terminology.  Often  the  purpose  of 
communication  is  to  clarify  or  acknowledge  the  receipt  of  information.  Feedback  involves  the 
giving,  seeking,  and  receiving  of  information  among  team  members.  Giving  feedback  refers  to 
providing  information  regarding  other  members’  performance.  Seeking  feedback  refers  to 
requesting  input  or  guidance  regarding  performance.  Receiving  feedback  refers  to  accepting 
positive  and  negative  information  regarding  performance.  Backup  behavior  involves  assisting 
the  performance  of  other  team  members.  This  implies  that  members  have  an  understanding  of 
other  members’  tasks.  It  also  implies  that  team  members  are  willing  and  able  to  provide  and  seek 
assistance  when  needed.  Monitoring  refers  to  observing  the  activities  and  performance  of  other 
team  members.  It  implies  that  team  members  are  individually  competent  and  that  they  may 
subsequently  provide  feedback  and  backup  behavior.  Coordination  refers  to  team  members 
executing  their  activities  in  a  timely  and  integrated  manner.  It  implies  that  the  performance  of 
some  team  members  influences  the  performance  of  other  team  members.  This  may  involve  an 
exchange  of  information  that  subsequently  influences  another  member’s  performance. 

The  emphasis  here  is  on  closed-loop  communication.  The  system  is  thus  comprised  of 
input,  throughput,  and  output  variables  through  which  communication  is  a  common  thread. 

Team  coordination  is  the  result  of  members  who  are  dedicated  to  the  team  task  providing, 
seeking,  and  receiving  feedback  to  support  each  other’s  efforts;  lending  backup  to  help  others 
perform  their  tasks;  and  continually  monitoring  the  team’s  performance.  These  throughput 
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variables  of  monitoring,  feedback,  and  backup  are  influenced  both  by  members’  orientation  or 
commitment  to  their  team  and  task,  and  the  structure  and  the  mission  of  the  team  as  dictated  by 
the  leader.  Finally,  all  team  processes  are  linked  through  sharing  information  or  communication. 

Morgan.  Glickman.  Woodard.  Blaiwes,  and  Salas  (1986):  TEAM  Model 

The  Team  Evolution  and  Maturation  (TEAM)  model  was  developed  as  part  of  a  project 
sponsored  by  the  Naval  Training  Systems  Center  (NTSC)  to  examine  the  factors  that  influence 
the  development  of  teamwork  during  training.  The  model  is  based  on  the  results  of  an  extensive 
literature  review  and  has  its  origins  in  an  open  systems  framework  of  organizational 
effectiveness. 

Morgan  et  al.  (1986)  adopted  a  developmental  view  of  teams.  The  TEAM  model 
combines  the  constructs  developed  by  Tuckman  (1965)  and  the  findings  of  Gersick  (1988)  to 
form  a  model  of  team  performance  that  predicts  the  stages  that  teams  go  through  before,  during, 
and  after  performance  of  a  task  (i.e.,  preforming,  forming,  storming,  norming,  performing-I, 
reforming,  performing-II,  conforming,  and  de-forming).  Morgan  and  his  colleagues  posit  that 
teams  evolve  through  this  series  of  stages.  Teams  may  begin  at  different  stages  of  development 
and  spend  different  amounts  of  time  in  the  various  stages.  Teams  do  not  necessarily  progress 
through  all  the  stages. 

All  teams  must,  however,  resolve  along  two  tracks  in  order  to  be  successful.  The  track  of 
“operational  team  skills  training”  represents  the  skills  needed  for  individual  team  members  to 
perform  their  respective  tasks.  The  track  of  “generic  team  skills  training”  includes  those 
activities  that  are  devoted  to  enhancing  the  quality  of  interactions,  relationships,  cooperation, 
communication,  and  coordination  of  team  members.  Specific  behaviors  included  in  this 
teamwork  track  include:  communication,  cooperation,  team  spirit  and  morale,  giving  suggestions 
or  criticism,  acceptance  of  suggestions  or  criticism,  coordination,  and  adaptability.  These  two 
tracks  must  converge  either  before  or  at  the  point  of  task  performance  in  order  for  the  team  to  be 
effective.  At  some  point  after  the  task  performance  these  two  tracks  again  diverge,  representing 
the  fact  that  all  teams  disperse  eventually. 

Olm  stead  £1 9921:  Model  of  Organizational  Competence 

Olmstead’s  work  has  been  aimed  specifically  at  understanding  the  dynamics  of  large 
military  organizations.  This  research  has  focused  on  issues  of  organizational  performance  and 
ways  of  improving  the  effectiveness  of  organizations.  His  work  investigating  the  functioning  of 
the  battle  staff  and  factors  leading  to  its  functional  integration  has  led  to  the  development  of  the 
Model  of  Organizational  Competence. 

The  Model  of  Organizational  Competence  describes  an  organization  as  a  homeostatic 
system  that  is  able  to  cope,  adapt,  and  become  better  integrated.  This  model  has  been  applied  to 
the  Army  battle  command,  where  the  battle  staff  receives  information  from  above,  works  on  the 
information  within  the  staff,  and  sends  information  out  to  effect  change.  Olmstead  defines 
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organizational  competence  as  the  functions,  or  processes,  required  by  organizational  systems  for 
effective  accomplishment  of  missions  or  objectives. 

According  to  Olmstead,  Organizational  Competence  is  concerned  with  the  quality  of 
performance  by  the  command-and-control  system  of  a  combat  unit.  The  command-and-control 
network  serves  as  the  brain  and  nervous  system  of  a  combat  unit,  acquiring  information  from 
various  sources,  collating  all  information,  making  decisions  concerning  actions  to  be  taken,  and 
sending  appropriate  instructions  and  directives  to  personnel  who  are  in  contact  with  opposing 
forces.  The  extent  to  which  this  system  functions  flexibly,  efficiently,  and  effectively 
determines,  in  large  part,  the  ability  of  the  unit  to  accomplish  its  tactical  objectives. 

Competence  depends  upon  skills  of  battle  staff  personnel  in  acquiring  and  interpreting 
information;  making  choices  concerning  to  whom  acquired  information  is  to  be  communicated, 
as  well  as  communicating  accurately  and  completely;  making  decisions  concerning  ways  to  cope 
with  unusual  or  unanticipated  situations;  and  executing  actions  deriving  from  such  decisions  — 
all  performed  at  high  levels  of  proficiency  and  coordination.  Some  technological  assists  may  be 
available,  such  as  data-processing  equipment,  electronic  surveillance  equipment,  and  highly 
sophisticated  communications  devices;  however,  the  payoff  in  competence  ultimately  reduces  to 
the  judgments  and  actions  of  key  personnel.  Of  equal  importance,  performance  of  the  processes 
is  a  team  product  and  much  of  the  quality  of process  performance  depends  upon  teamwork  and 
the  coordination  of  separate  responsibilities  and  activities. 

Olmstead  contends  that  equal  to  the  skills  of  individuals  is  what  he  terms  “the  integration 
of  structure  and  function.”  This  means  that  the  positions,  roles,  and  functions  that  make  up  an 
organizational  system  must  fit  together  and  support  each  other  in  their  respective  activities.  In 
short,  integration  of  a  battle  staff,  with  the  resulting  teamwork,  is  essential.  If  integration  of 
structure  and  function  does  not  occur,  missed  signals,  aborted  decisions,  overlooked  intelligence, 
and  activities  at  cross-purposes  may  be  the  result.  In  the  extreme,  loss  of  integration  may 
produce  a  collapse  of  essential  functions,  which  can  threaten  survival  of  the  unit. 

The  Model  of  Organizational  Competence  presents  three  aspects  of  organizational 
competence.  Reality  testing  is  concerned  with  determining  the  real  properties  of  an  environment. 
How  well  is  the  team  able  to  assess  the  operational  situation?  Adaptability  is  concerned  with 
learning  through  experience.  This  refers  to  the  team’s  ability  to  react  flexibly  to  changing 
requirements.  Integration  is  concerned  with  unifying  the  actions  of  team  members,  or  the  team’s 
ability  to  maintain  itself  under  times  of  stress.  Olmstead  claims  that  it  is  critical  that  a  team 
maintains  adequate  performance  in  each  of  these  components.  This  speaks  against  trying  to 
maintain  a  rigid  top-down  control  of  organizations  in  a  dynamic  environment  that  demands 
responsiveness.  An  effective  team  must  be  able  to  assess  its  own  environment,  adapt  quickly  to 
changing  requirements,  and  maintain  cohesiveness  of  all  members  and  sub-units. 

In  addition  to  these  three  aspects,  seven  processes  of  organizational  competence  are 
proposed.  These  seven  processes  are  summarized  in  Table  4.  Sensing  refers  to  the  process  by 
which  the  team  acquires  and  interprets  information  concerning  the  state  of,  or  events  occurring 
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in,  the  environment.  Communicating  information  is  about  getting  the  right  information  to  the 
right  people.  -This  process  involves  the  initial  transmittal  of  information  by  those  who  have 
sensed  it,  to  dissemination  of  the  information  throughout  the  organization.  Feedback  is  the 
process  of  assessing  and  evaluating  the  effects  of  prior  action.  This  involves  further  sensing  of 
the  external  and  internal  environments.  Decision  making  includes  those  activities  leading  to  the 
conclusion  that  some  action  should  be  taken.  Communicating  implementation  refers  to  the 
process  whereby  decisions  and  resulting  requirements  are  communicated  to  those  individuals 
who  must  implement  them.  This  includes  achieving  clarification  of  orders  through  discussion 
and  interpretation.  Coping  actions  are  those  activities  involving  direct  action  against  external 
and  internal  environments.  This  is  the  ultimate  determinant  of  effectiveness:  the  effect  of  the 
organization  upon  the  target  environment.  Stabilizing  is  the  process  of  taking  action  to  adjust 
internal  operations  to  maintain  stability  in  the  face  of  potential  disruption. 


Table  4.  Criteria  for  assessing  quality  of  process  performance.  (Olmstead,  1992) 


Sensing 

a.  Accurate  detection  of  all  available  information 

b.  Correct  interpretation  (attachment  of  correct  meaning)  of  all  detected  information,  to 
include  appropriate  weighing  of  its  importance 

c.  Accurate  discrimination  between  relevant  and  irrelevant  information 

d.  Attempts  to  obtain  information  are  relevant  to  mission,  task,  or  problem 

e.  Sensing  activities  are  timely  in  relation  to  information  requirements  and  the  tactical 
situation  of  the  moment 

f.  Internal  processing  and  recording  of  information  provides  ready  availability  to  users 

Communicating  Information 

a.  Accuracy  of  transmission  of  available  information  . 

b.  Sufficiently  complete  to  transmit  full  and  accurate  understanding  to  receivers  of 
communications 

c.  Timeliness  appropriate  to  unit  requirements 

d.  Correct  choice  of  recipients;  everyone  who  needs  information  receives  it 

e.  Whether  message  should  have  been  communicated 

Decision  Making 

a.  Adequacy  -  Was  the  decision  adequately  correct  in  view  of  circumstances  and 
information  available  to  the  decision  maker? 

b.  Appropriateness  -  Was  the  decision  timely  in  view  of  the  information  available  to  the 
decision  maker? 

c.  Completeness  -  Did  the  decision  take  into  account  all  or  most  contingencies, 
alternatives,  and  possibilities? 


48 


Stabilizing 


a.  Adequacy  -  Action  is  correct  in  view  of  the  operational  situation  and  conditions 
which  the  action  is  intended  to  change  or  overcome 

b.  Appropriateness  -  Timing  is  appropriate  in  view  of  the  situation,  conditions,  and 
intended  effects.  Choice  of  target  of  the  action  is  appropriate 

c.  Completeness  -  Action  fully  meets  the  requirements  of  the  situation 

Communicating  Implementation 

a.  Accuracy  of  transmission  of  instructions 

b.  Sufficient  completeness  to  transmit  adequate  and  full  understanding  of  actions 
required 

c.  Timely  transmission  in  view  of  both  available  information  and  the  action 
requirements  of  the  participants 

d.  Transmission  to  appropriate  recipients 

e.  “Discussion  or  interpretation”  is  efficient,  relevant,  and  achieves  its  purpose 

f.  Whether  message  should  have  been  communicated 

Coping  Actions 

a.  Correctness  of  actions  in  view  of  both  the  current  operational  circumstances  and  the 
decision  or  order  from  which  the  action  derives 

b.  Timeliness  of  the  action  in  view  of  both  operational  circumstances  and  the  decision  or 
order  from  which  the  action  derives 

c.  Correctness  of  choice  of  target  of  the  action 

d.  Adequacy  of  execution  of  action 

Feedback 


a.  Correctness  of  the  decision  and  action  to  obtain  feedback  in  view  of  operational 
circumstances  the  preceding  actions  whose  results  are  being  evaluated,  and  current 
information  requirements 

b.  Timeliness  of  the  feedback  decision  and  action 

c.  Correctness  of  choice  of  target(s)  of  the  action 

d.  Appropriate  use  of  feedback  information  in  new  actions,  decisions,  and  plans 
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Helmreich  and  Foushee  ('1993'):  Crew  Resource  Management 

Helmreich  and  Foushee’s  (1993)  work  utilizes  an  interpersonal  view  of  teams.  Crew 
Resource  Management  (CRM)  grew  out  of  concerns  for  aviation  safety  during  the  1970’s. 
Experimental  evidence  collected  by  Ruffel  Smith  (1979)  first  led  to  the  identification  of  resource 
management  behavior  as  an  important  variable  in  aviation  safety  and  to  the  need  for  related 
behavioral  skills  training.  In  a  full  mission  simulation,  Ruffell  Smith  found  that  a  crew’s 
effectiveness  in  identifying  and  utilizing  the  human  material  resources  available  influenced  how 
safely  and  effectively  the  crew  handled  problem  situations. 

Robert  Helmreich  and  his  colleagues  at  the  University  of  Texas  and  NASA  Ames 
Research  Center  initiated  a  program  of  research  to  investigate  the  foundations  of  flight  crew 
behavior,  to  develop  measures  of  crew  performance,  and  to  develop  training  designed  to  enhance 
aviation  safety  by  enhancing  pilots’  abilities  to  work  as  effective  teams.  Wiener,  Kanki,  and 
Helmreich  (1993)  provide  the  most  recent  comprehensive  review  of  the  CRM  field. 

Helmreich  and  Foushee  (1993)  developed  CRM  concepts  within  the  boundaries  of  a 
three-factor  model  of  the  determinants  of  group  performance.  This  model  defines  three  major 
components  of  group  behavior:  input  factors,  crew  performance  or  group  process  factors,  and 
outcome  factors.  Input  factors  include  characteristics  of  individuals,  groups,  organizations,  and 
the  operational  environment.  Group  process  factors  include  the  nature  and  quality  of  interactions 
among  group  members.  Outcome  factors  include  primary  outcomes  such  as  safety  and  efficiency 
of  operations  as  well  as  secondary  outcomes  such  as  member  satisfaction,  motivation,  and 
attitudes.  The  underlying  assumption  of  the  model  is  that  input  factors  provide  the  framework 
and  determine  the  nature  of  group  processes  that  lead  to  the  various  outcomes.  A  central  feature 
of  the  model  is  the  presence  of  feedback  loops  between  the  components.  Outcomes  may  change 
the  components  of  input  factors,  and  these  changes  may  alter  subsequent  group  processes  and 
outcomes.  In  addition,  outcomes  may  alter  group  processes  directly  without  affecting  input 
factors. 


Helmreich  uses  this  three-factor  model  as  a  framework  for  studying  the  CRM  functions  of 
team  performance.  He  describes  CRM  performance  with  a  three-layer  hierarchical  model.  At 
the  highest  level  are  three  clusters  of  observable  CRM  behaviors:  Communications  and  Decision 
Making  Tasks,  Team  Formation  and  Management  Tasks,  and  Situation  Awareness  and  Workload 
Management  Tasks. 

Communications  Process  and  Decision  Tasks  concern  how  the  team  members 
communicate  with  each  other  and  the  decision  processes  that  they  employ.  These  are  both  prime 
determinants  of  crew  performance.  This  cluster  of  tasks  comprises  the  observable  behaviors: 
Briefings,  Inquiry/Assertion,  Self  Critique,  Communications,  and  Decisions. 

Team  Formation  and  Management  Tasks  concern  the  formation  of  the  crew  as  a  team. 
Early  in  the  formation  process,  teams  establish  communication  and  interaction  patterns  that 
persist  throughout  the  team  activity.  These  patterns  can  lead  to  either  effective  or  ineffective 
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team  performance.  This  cluster  of  tasks  comprises  the  behaviors:  Leadership,  Task  Concern,  and 
Group  Climate. 

The  third  cluster,  Situational  Awareness  and  Workload  Management  Tasks,  addresses 
concerns  about  the  crew’s  awareness  of  operational  conditions  and  contingencies  and  how  the 
crew  distributes  tasks  to  avoid  overloading  any  crew  member.  The  observable  behaviors  include: 
Preparation,  Planning,  Vigilance,  Workload  Distribution,  Task  Prioritization,  and  Distraction 
Avoidance. 

The  lowest  level  of  the  model  of  CRM  comprises  specific  behavioral  markers.  Helmreich 
and  his  colleagues  (1993)  have  identified  behavioral  markers  for  each  of  the  behaviors  described 
above.  Trained  evaluators  employ  expert  rating  scales  to  assess  performance  on  each  of  these 
markers,  and  use  these  ratings  to  assess  performance  on  each  of  the  behaviors.  Such  assessments 
are  then  used  to  focus  debriefings  following  training  sessions. 

Additional  Accounts  of  Team  Mind 

There  are  other  accounts  of  the  team  mind  that  have  not  been  developed  as  fully  as  these 
mentioned  above.  We  list  these  here  because  they  offer  different  perspectives  that  may  have 
some  value  in  studying  Information  Dominance. 

The  concept  of  shared  mental  models,  developed  and  applied  by  Orasanu  and  Salas 
(1993),  Cannon-Bowers,  Salas,  and  Converse  (1992)  and  others,  is  being  used  in  the  Navy  to 
train  teams.  Prince,  Chidester,  Cannon-Bowers,  and  Bowers  (1993)  have  developed  several 
well-accepted  team  training  programs  for  the  Navy.  Helmreich  (1986)  has  expanded  his  initial 
Crew  Resource  Management  program  from  use  in  commercial  aviation  to  a  wide  array  of 
domains. 

Salas,  Cannon-Bowers,  and  Johnston  (1997)  have  concluded  that  expert  teams  can  be 
developed  by: 

•  fostering  shared  or  compatible  mental  models  of  the  task  and  of  the  roles  of  each  team 
member, 

•  training  the  team  members  on  teamwork  skills  such  as  situation  awareness, 
leadership,  and  compensatory  behavior, 

•  providing  experience  for  teams  to  function  under  the  types  of  stressful  conditions  they 
will  encounter,  by  cross-training  (letting  the  team  members  practice  on  the  roles  and 
tasks  of  others), 

•  showing  leaders  how  to  maintain  shared  situation  awareness. 

Hinsz,  Tindale  &  Vollrath  (1997)  have  reviewed  a  variety  of  approaches  that  treat  groups 
as  information  processors,  a  central  tenet  of  the  “team  mind”  framework.  They  have  examined 
research  on  attention,  encoding,  storage,  retrieval,  processing,  response,  feedback,  and  learning  in 
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small  interacting  groups.  These  research  efforts  are  also  relevant  to  our  understanding  of  team 
mind  phenomena.  Along  these  lines,  Endsley  and  Robertson  (1996)  have  examined  team 
situation  awareness  in  aviation  maintenance  settings. 
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